Discussion:
Snake farm
Neal Norwitz
2002-11-02 23:51:01 UTC
Permalink
As some of you may have noticed, I recently checked in a bunch of
fixes that deal with portability. I have been granted access to the
snake farm, which allows me to test python on different architectures.

The snake farm architectures include:

HPUX 11
AIX 4.2 & 4.3
Linux 2.4 & 2.2/Alpha
Solaris 8
SunOS 4.1.1

I am mostly testing 2.3, but can test 2.2 as well.

Right now, there is no easy way to tell if a bug on SF is specific to
a particular OS/architecture. There are already Macintosh and Windows
categories, perhaps a UNIX category should be added? In the meantime,
feel free to assign UNIX platform specific problems to me. I know
there are some FreeBSD problems. There is a FreeBSD machine in the SF
compile farm (ssh to compile.sf.net). Just in case someone wants to
try to fix the FreeBSD bugs. :-)

Neal
Anders Qvist
2002-11-03 08:51:12 UTC
Permalink
Post by Neal Norwitz
I am mostly testing 2.3, but can test 2.2 as well.
The PBF people have also requested this.
Post by Neal Norwitz
Right now, there is no easy way to tell if a bug on SF is specific to
a particular OS/architecture. There are already Macintosh and Windows
categories, perhaps a UNIX category should be added? In the meantime,
feel free to assign UNIX platform specific problems to me. I know
there are some FreeBSD problems. There is a FreeBSD machine in the SF
compile farm (ssh to compile.sf.net). Just in case someone wants to
try to fix the FreeBSD bugs. :-)
The lack of a FreeBSD-machine in snake farm is only due to laziness.
There is hardware available. There might even be a FreeBSD/intel
machine hidden away in the computer hall, just that noone have found
it yet :). I'll see what I can do (picture week-long expedition
through a vast maze of 19"-racks, huge cabinets and long-decommisioned
Crays).
--
Anders "Quest" Qvist

"We've all heard that a million monkeys banging on a million typewriters
will eventually reproduce the entire works of Shakespeare. Now, thanks
to the Internet, we know this is not true." -- Robert Wilensky
Anders Qvist
2002-11-03 09:00:53 UTC
Permalink
Post by Anders Qvist
Post by Neal Norwitz
I am mostly testing 2.3, but can test 2.2 as well.
The PBF people have also requested this.
Which CVS branch do I want to test (there are a few to choose from)?
--
Anders "Quest" Qvist

"We've all heard that a million monkeys banging on a million typewriters
will eventually reproduce the entire works of Shakespeare. Now, thanks
to the Internet, we know this is not true." -- Robert Wilensky
Neal Norwitz
2002-11-03 22:08:16 UTC
Permalink
Post by Anders Qvist
Post by Anders Qvist
Post by Neal Norwitz
I am mostly testing 2.3, but can test 2.2 as well.
The PBF people have also requested this.
Which CVS branch do I want to test (there are a few to choose from)?
I typically do something like this:

cd python/dist
cp -pr src 2.2
cd 2.2
cvs upd -dP -r release22-maint

The CVS tag is release22-maint.

Neal
Hye-Shik Chang
2002-11-04 19:29:37 UTC
Permalink
Post by Neal Norwitz
As some of you may have noticed, I recently checked in a bunch of
fixes that deal with portability. I have been granted access to the
snake farm, which allows me to test python on different architectures.
HPUX 11
AIX 4.2 & 4.3
Linux 2.4 & 2.2/Alpha
Solaris 8
SunOS 4.1.1
I am mostly testing 2.3, but can test 2.2 as well.
Right now, there is no easy way to tell if a bug on SF is specific to
a particular OS/architecture. There are already Macintosh and Windows
categories, perhaps a UNIX category should be added? In the meantime,
feel free to assign UNIX platform specific problems to me. I know
there are some FreeBSD problems. There is a FreeBSD machine in the SF
compile farm (ssh to compile.sf.net). Just in case someone wants to
try to fix the FreeBSD bugs. :-)
BTW, Python-CVS can't be compiled FreeBSD-CURRENT, nowadays.
Because FreeBSD-CURRENT hides non-POSIX stuffs on _POSIX_C_SOURCE mode,
Modules/posixmodule.c fails to find chroot, minor, major, makedev and etc.
This patch enables python builds on FreeBSD-CURRENT but I can't
find another generous workaround.

Index: configure.in
===================================================================
RCS file: /cvsroot/python/python/dist/src/configure.in,v
retrieving revision 1.362
diff -c -r1.362 configure.in
*** configure.in 2 Nov 2002 16:58:05 -0000 1.362
--- configure.in 4 Nov 2002 19:12:11 -0000
***************
*** 24,49 ****
AC_SUBST(SOVERSION)
SOVERSION=1.0

- # The later defininition of _XOPEN_SOURCE disables certain features
- # on Linux, so we need _GNU_SOURCE to re-enable them (makedev, tm_zone).
- AC_DEFINE(_GNU_SOURCE, 1, [Define on Linux to activate all library features])
-
- # The definition of _GNU_SOURCE potentially causes a change of the value
- # of _XOPEN_SOURCE. So define it only conditionally.
- AH_VERBATIM([_XOPEN_SOURCE],
- [/* Define on UNIX to activate XPG/5 features. */
- #ifndef _XOPEN_SOURCE
- # define _XOPEN_SOURCE 500
- #endif])
- AC_DEFINE(_XOPEN_SOURCE, 500)
-
- # On Tru64 Unix 4.0F, defining _XOPEN_SOURCE also requires definition
- # of _XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE, or else several APIs
- # are not declared. Since this is also needed in some cases for HP-UX,
- # we define it globally.
- AC_DEFINE(_XOPEN_SOURCE_EXTENDED, 1, Define to activate Unix95-and-earlier features)
- AC_DEFINE(_POSIX_C_SOURCE, 199506L, Define to activate features from IEEE Stds 1003.{123}-1995)
-
# Arguments passed to configure.
AC_SUBST(CONFIG_ARGS)
CONFIG_ARGS="$ac_configure_args"
--- 24,29 ----
***************
*** 131,136 ****
--- 111,144 ----
fi
AC_MSG_RESULT($MACHDEP)

+ # The later defininition of _XOPEN_SOURCE disables certain features
+ # on Linux, so we need _GNU_SOURCE to re-enable them (makedev, tm_zone).
+ AC_DEFINE(_GNU_SOURCE, 1, [Define on Linux to activate all library features])
+
+ # The definition of _GNU_SOURCE potentially causes a change of the value
+ # of _XOPEN_SOURCE. So define it only conditionally.
+ AH_VERBATIM([_XOPEN_SOURCE],
+ [/* Define on UNIX to activate XPG/5 features. */
+ #if !defined(_XOPEN_SOURCE) && !defined(__FreeBSD__)
+ # define _XOPEN_SOURCE 500
+ #endif])
+
+ case $MACHDEP in
+ # FreeBSD hides non-POSIX functions on POSIX-compatible mode.
+ freebsd*)
+ ;;
+ *)
+ AC_DEFINE(_XOPEN_SOURCE, 500)
+
+ # On Tru64 Unix 4.0F, defining _XOPEN_SOURCE also requires definition
+ # of _XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE, or else several APIs
+ # are not declared. Since this is also needed in some cases for HP-UX,
+ # we define it globally.
+ AC_DEFINE(_XOPEN_SOURCE_EXTENDED, 1, Define to activate Unix95-and-earlier features)
+ AC_DEFINE(_POSIX_C_SOURCE, 199506L, Define to activate features from IEEE Stds 1003.{123}-1995)
+ ;;
+ esac
+
# checks for alternative programs
AC_MSG_CHECKING(for --without-gcc)
AC_ARG_WITH(gcc,


But, this build gets segfault on installing.

... make install ...
./python -E ./setup.py install --prefix=/home/perky/test --install-scripts=/home/perky/test/bin --install-platlib=/home/perky/test/lib/python2.3/lib-dynload
running install
running build
running build_ext
Segmentation fault (core dumped)
*** Error code 139

#0 PyObject_Free (p=0x800) at Objects/obmalloc.c:713
713 if (ADDRESS_IN_RANGE(p, pool->arenaindex)) {
(gdb) bt
#0 PyObject_Free (p=0x800) at Objects/obmalloc.c:713
#1 0x080e0533 in function_call (func=0x8210b1c, arg=0x82129ac, kw=0x821ebdc)
at Objects/funcobject.c:481
#2 0x08059564 in PyObject_Call (func=0x0, arg=0x82129ac, kw=0x821ebdc) at Objects/abstract.c:1688
#3 0x0809ea95 in ext_do_call (func=0x8210b1c, pp_stack=0xbfbfddbc, flags=0, na=1, nk=0)
at Python/ceval.c:3438
#4 0x0809d14f in eval_frame (f=0x81f500c) at Python/ceval.c:2031
.....
Post by Neal Norwitz
Neal
_______________________________________________
Python-Dev mailing list
http://mail.python.org/mailman/listinfo/python-dev
Regards,
--
Hye-Shik Chang <***@fallin.lv>
Yonsei University, Seoul
^D
Martin v. Loewis
2002-11-04 22:06:31 UTC
Permalink
Post by Hye-Shik Chang
BTW, Python-CVS can't be compiled FreeBSD-CURRENT, nowadays.
Because FreeBSD-CURRENT hides non-POSIX stuffs on _POSIX_C_SOURCE
mode, Modules/posixmodule.c fails to find chroot, minor, major,
makedev and etc. This patch enables python builds on
FreeBSD-CURRENT but I can't find another generous workaround.
I don't like this approach. For -CURRENT, I would outright reject any
such patch; there should be a way to enable extensions even if
_POSIX_C_SOURCE is defined. Perhaps they reconsider until they release
the system.

OTOH, why absence of chroot a problem? Should not HAVE_CHROOT be
undefined if chroot is hidden?

Regards,
Martin
Marc Recht
2002-11-10 17:23:17 UTC
Permalink
Post by Martin v. Loewis
I don't like this approach. For -CURRENT, I would outright reject any
such patch; there should be a way to enable extensions even if
A somewhat simpler solution/work-around would be to define __BSD_VISIBLE
(patch attached). But the cleanest way would be to not define
_XOPEN_SOURCE, XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE on FreeBSD 5.
Post by Martin v. Loewis
_POSIX_C_SOURCE is defined. Perhaps they reconsider until they release
the system.
Unlikely. The release is scheduled for the 20th of November.
Post by Martin v. Loewis
OTOH, why absence of chroot a problem? Should not HAVE_CHROOT be
undefined if chroot is hidden?
It isn't.

Regards,
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-10 20:40:55 UTC
Permalink
Post by Marc Recht
Post by Martin v. Loewis
I don't like this approach. For -CURRENT, I would outright reject any
such patch; there should be a way to enable extensions even if
A somewhat simpler solution/work-around would be to define __BSD_VISIBLE
(patch attached). But the cleanest way would be to not define
_XOPEN_SOURCE, XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE on FreeBSD 5.
Notice that issues are different on the various BSDs.

I think of Python on Unix as "POSIX+Extensions". On all POSIX systems,
_XOPEN_SOURCE should be defined. If additional defines are needed to
activate extensions, we should define them.

If FreeBSD has no mechanisms to request extensions other than defining
__BSD_VISIBLE, we should define it. Before doing so, I'd like to know
what Python features would need that.

Please don't post patches to python-dev.
Post by Marc Recht
Post by Martin v. Loewis
OTOH, why absence of chroot a problem? Should not HAVE_CHROOT be
undefined if chroot is hidden?
It isn't.
Ok, then this needs to be investigated. This is a clear bug.

Regards,
Martin
Marc Recht
2002-11-10 21:10:57 UTC
Permalink
Post by Martin v. Loewis
Post by Marc Recht
(patch attached). But the cleanest way would be to not define
_XOPEN_SOURCE, XOPEN_SOURCE_EXTENDED and _POSIX_C_SOURCE on FreeBSD 5.
Notice that issues are different on the various BSDs.
I know. And if there are issues they should be resolved for each BSD.
Post by Martin v. Loewis
I think of Python on Unix as "POSIX+Extensions". On all POSIX systems,
_XOPEN_SOURCE should be defined. If additional defines are needed to
activate extensions, we should define them.
The clean way on FreeBSD is then _not_ to define the above defines. We
then get everything we want.
If you define _POSIX_C_SOURCE you get _POSIX_C_SOURCE. Not more, not
less..
Post by Martin v. Loewis
If FreeBSD has no mechanisms to request extensions other than defining
__BSD_VISIBLE, we should define it. Before doing so, I'd like to know
what Python features would need that.
Setting __BSD_VISIBLE is rather a hack and shouldn't be done..
Please have a look at the patch, which I submitted on SourceForge. (It's somewhat like
Hye-Shik Chang patch.)
Post by Martin v. Loewis
Please don't post patches to python-dev.
Sorry..

Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-10 22:00:14 UTC
Permalink
Post by Marc Recht
The clean way on FreeBSD is then _not_ to define the above defines. We
then get everything we want.
If you define _POSIX_C_SOURCE you get _POSIX_C_SOURCE. Not more, not
less..
Can you elaborate? We also define _XOPEN_SOURCE. So we are entitled to
get all functions defined for UNIX.

I consider it a bug that FreeBSD does not provide a mode for
"Conforming XSI Application Using Extensions", according to

http://www.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap02.html
Post by Marc Recht
Setting __BSD_VISIBLE is rather a hack and shouldn't be done..
If it is a hack to work around a bug in FreeBSD, then I think it is
acceptable.
Post by Marc Recht
Please have a look at the patch, which I submitted on SourceForge.
It is way too large to be acceptable, and takes a "I care only about
one system" position. Try writing your code in a way so that it
simultaneously works with many systems, instead of special-casing each
system individually.

For example, why is it necessary to move the enable-framework
processing?

Regards,
Martin
Marc Recht
2002-11-10 22:31:35 UTC
Permalink
Post by Martin v. Loewis
Can you elaborate? We also define _XOPEN_SOURCE. So we are entitled to
get all functions defined for UNIX.
Hmm.. I'm not that standards guy and didn't read that out of that
document. It enable the extensions. (Though I'm not that sure which that
are..)
Post by Martin v. Loewis
I consider it a bug that FreeBSD does not provide a mode for
"Conforming XSI Application Using Extensions", according to
FreeBSD does support that. _XOPEN_SOURCE = 500 means:
#define __XSI_VISIBLE 500
#undef _POSIX_C_SOURCE
#define _POSIX_C_SOURCE 199506
Post by Martin v. Loewis
http://www.opengroup.org/onlinepubs/007904975/basedefs/xbd_chap02.html
Post by Marc Recht
Setting __BSD_VISIBLE is rather a hack and shouldn't be done..
If it is a hack to work around a bug in FreeBSD, then I think it is
acceptable.
I still not think that's a bug in FreeBSD. But I send you sys/cdefs.h
and unistd.h, so you can have a look for yourself. Maybe I'm getting
something wrong here. IMO it's just a strict interpretation of the
standard.
Post by Martin v. Loewis
It is way too large to be acceptable, and takes a "I care only about
Why? There are only two actual changes the case for FreeBSD and the
addition of _THREAD_SAFE to the CFLAGS if Python is compiled with
threads. The rest just moves the MACHDEP stuff to the top of
configure.in.
Post by Martin v. Loewis
one system" position. Try writing your code in a way so that it
I don't think so.. Sometimes systems need special treatment...
Post by Martin v. Loewis
simultaneously works with many systems, instead of special-casing each
system individually.
I didn't saw an other way to do it. The defines were always set and that
just doesn't work for FreeBSD.
Post by Martin v. Loewis
For example, why is it necessary to move the enable-framework
processing?
To define MACHDEP before the XOPEN/POSIX stuff, because of the FreeBSD
case.

Regards
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-11 05:57:48 UTC
Permalink
Post by Marc Recht
Post by Martin v. Loewis
I consider it a bug that FreeBSD does not provide a mode for
"Conforming XSI Application Using Extensions", according to
#define __XSI_VISIBLE 500
#undef _POSIX_C_SOURCE
#define _POSIX_C_SOURCE 199506
So how do I request extensions on that system, beyond the functions
defined in XPG/5 or XPG/6?
Post by Marc Recht
Post by Martin v. Loewis
If it is a hack to work around a bug in FreeBSD, then I think it is
acceptable.
I still not think that's a bug in FreeBSD. But I send you sys/cdefs.h
and unistd.h, so you can have a look for yourself. Maybe I'm getting
something wrong here. IMO it's just a strict interpretation of the
standard.
I have looked at them before, hence my claim that I think FreeBSD has
a bug here.
Post by Marc Recht
Why? There are only two actual changes the case for FreeBSD and the
addition of _THREAD_SAFE to the CFLAGS if Python is compiled with
threads. The rest just moves the MACHDEP stuff to the top of
configure.in.
I misread the patch: That it moves the MACHDEP stuff is not visible.
Instead, it moves the --enable-framework stuff downwards.

Still, you add a case with "FreeBSD", and "rest of the world", the
"rest of the world" case being where _GNU_SOURCE, _XOPEN_SOURCE,
_XOPEN_SOURCE_EXTENDED, and _POSIX_C_SOURCE is defined. Why does it
hurt if either _GNU_SOURCE or _XOPEN_SOURCE_EXTENDED would be defined
on FreeBSD?
Post by Marc Recht
Post by Martin v. Loewis
one system" position. Try writing your code in a way so that it
I don't think so.. Sometimes systems need special treatment...
Only if all other options have been exhausted. What problems occur if
_XOPEN_SOURCE is defined?
Post by Marc Recht
Post by Martin v. Loewis
simultaneously works with many systems, instead of special-casing each
system individually.
I didn't saw an other way to do it. The defines were always set and that
just doesn't work for FreeBSD.
Did you try defining _GNU_SOURCE and _XOPEN_SOURCE_EXTENDED? I can see
that you don't want _POSIX_C_SOURCE to be defined, either.

Regards,
Martin
Marc Recht
2002-11-11 10:25:20 UTC
Permalink
Post by Martin v. Loewis
So how do I request extensions on that system, beyond the functions
defined in XPG/5 or XPG/6?
The used standard is defined by_POSIX_SOURCE, _POSIX_C_SOURCE,
_ANSI_SOURCE and _C99_SOURCE. In that order (sys/cdefs.h). These set
the internal defines __POSIX_VISIBLE, __XSI_VISIBLE, __BSD_VISIBLE,
__ISO_C_VISIBLE according to the given standard. If nothing is given
then
#define __POSIX_VISIBLE 200112
#define __XSI_VISIBLE 600
#define __BSD_VISIBLE 1
#define __ISO_C_VISIBLE 1999
is set. In this environment is everything we want.
Post by Martin v. Loewis
I have looked at them before, hence my claim that I think FreeBSD has
There have been a lot of changes between September and late October.
Post by Martin v. Loewis
a bug here.
I'm still not convinced that FreeBSD's behaviour is a bug. IMO it's only
strict. If you define a standard you get, if not you get all.

Although I'm quite sure the FreeBSD community doesn't like it I'll make
a patch against sys/defs.h to put a define _BSD_SOURCE at the top of the
chain which sets the default environment and post it to the
freebsd-current discussion list.
Post by Martin v. Loewis
Still, you add a case with "FreeBSD", and "rest of the world", the
I'm quite sure it's an object to be extended.. There are some strange
systems out there which all need special treatment..
Post by Martin v. Loewis
"rest of the world" case being where _GNU_SOURCE, _XOPEN_SOURCE,
_XOPEN_SOURCE_EXTENDED, and _POSIX_C_SOURCE is defined. Why does it
hurt if either _GNU_SOURCE or _XOPEN_SOURCE_EXTENDED would be defined
on FreeBSD?
They both don't hurt. I only moved them into the "*" case, because
they're not needed for the FreeBSD case.
Post by Martin v. Loewis
Only if all other options have been exhausted. What problems occur if
_XOPEN_SOURCE is defined?
_XOPEN_SOURCE sets _POSIX_C_SOURCE.
Post by Martin v. Loewis
Did you try defining _GNU_SOURCE and _XOPEN_SOURCE_EXTENDED? I can see
It's no problem to define them. They're just not needed.
Post by Martin v. Loewis
that you don't want _POSIX_C_SOURCE to be defined, either.
If it's set then we don't get __BSD_VISIBLE.

Regards,
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-11 12:24:25 UTC
Permalink
Post by Marc Recht
Although I'm quite sure the FreeBSD community doesn't like it I'll make
a patch against sys/defs.h to put a define _BSD_SOURCE at the top of the
chain which sets the default environment and post it to the
freebsd-current discussion list.
Could you please call it __EXTENSIONS__? Many of the functions that
fall under it are not BSD specific, and _BSD_SOURCE selects the
favour-bsd API on other systems.

_BSD_SOURCE would be fine for things that are traditionally in BSD,
but have been superceded by competing POSIX API. In that case, Python
would want to use the POSIX API. So would would not want to define
_BSD_SOURCE (just as we don't want to define _OSF_SOURCE and
_GNU_SOURCE, but cannot avoid doing so).
Post by Marc Recht
I'm quite sure it's an object to be extended.. There are some strange
systems out there which all need special treatment..
... or just cannot be supported. Please see PEP 11, we'll be phasing
out support for a number of such systems. It's just not worth the
pain. Python works on Posix systems, and uses extensions where
available. It does not work on strange systems; tough luck for users
of such systems (we have yet to hear from a DYNIX user who wants to
run Python 2.2).
Post by Marc Recht
Post by Martin v. Loewis
Only if all other options have been exhausted. What problems occur if
_XOPEN_SOURCE is defined?
_XOPEN_SOURCE sets _POSIX_C_SOURCE.
And why is that a problem?

Regards,
Martin
Marc Recht
2002-11-11 17:07:10 UTC
Permalink
Post by Martin v. Loewis
Could you please call it __EXTENSIONS__? Many of the functions that
fall under it are not BSD specific, and _BSD_SOURCE selects the
favour-bsd API on other systems.
The responsible person seems to be against a solution like this. But, in
the discussion it turns out that you are right and there some bugs in
FreeBSD's unistd.h. The fixes will (probably) committed today. But
there are some issues left mostly in the RPC and socket code, which need
__BSD_VISIBLE. Some problems are left, because some functions (like
ftello) are only defined at a higher POSIX level.
Post by Martin v. Loewis
out support for a number of such systems. It's just not worth the
pain. Python works on Posix systems, and uses extensions where
available. It does not work on strange systems; tough luck for users
of such systems (we have yet to hear from a DYNIX user who wants to
run Python 2.2).
I'm not talking about strange systems. Although I've not tested it
BSD/OS comes to mind or NetBSD/OpenBSD.
Post by Martin v. Loewis
Post by Marc Recht
Post by Martin v. Loewis
Only if all other options have been exhausted. What problems occur if
_XOPEN_SOURCE is defined?
_XOPEN_SOURCE sets _POSIX_C_SOURCE.
And why is that a problem?
In that case __BSD_VISIBLE is not set.

Regards,
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-11 18:25:55 UTC
Permalink
Post by Marc Recht
The responsible person seems to be against a solution like this. But, in
the discussion it turns out that you are right and there some bugs in
FreeBSD's unistd.h. The fixes will (probably) committed today. But
there are some issues left mostly in the RPC and socket code, which need
__BSD_VISIBLE.
Please be as precise as possible when talking about this stuff. What
issues precisely?

I'm not going to accept patches for platform-specific changes just
because a user said that these changes are needed "to make it
work". Instead, I want to record the *precise* cause that triggered
these changes, so that others can re-evaluate whether they are still
needed five years from now.
Post by Marc Recht
Some problems are left, because some functions (like ftello) are
only defined at a higher POSIX level.
Why is that a problem?
Post by Marc Recht
Post by Martin v. Loewis
Post by Marc Recht
Post by Martin v. Loewis
Only if all other options have been exhausted. What problems occur if
_XOPEN_SOURCE is defined?
_XOPEN_SOURCE sets _POSIX_C_SOURCE.
And why is that a problem?
In that case __BSD_VISIBLE is not set.
And why is that a problem?

This is exhausting.

Regards,
Martin
Marc Recht
2002-11-11 19:10:38 UTC
Permalink
Post by Martin v. Loewis
Please be as precise as possible when talking about this stuff. What
issues precisely?
Sorry. Trying to. I've posted more detail (including a build log) on SourceForge.
But again the major problem is certain defines and typedefs like PF_INET
or u_char, u_int (see SourceForge) are only defined in the __BSD_VISIBLE
case. These defines typedefs are needed by certain Python modules like
sockemodule, nismodule.
Post by Martin v. Loewis
I'm not going to accept patches for platform-specific changes just
because a user said that these changes are needed "to make it
I also a friend of generic/standard solutions,I don't see one here. I
would be more than happy proved to be wrong...
You say you want no platform specific stuff in configure and the person
responsible for the standard stuff on the FreeBSD side says:
"The whole point of the standards constants is to specify a strict
environment. If you want a BSD environment don't specify a particular
standard, it's simple.".
Post by Martin v. Loewis
work". Instead, I want to record the *precise* cause that triggered
these changes, so that others can re-evaluate whether they are still
needed five years from now.
I understand that, but this is a build fix. So the evaluation is quite
simple..
Post by Martin v. Loewis
Post by Marc Recht
Some problems are left, because some functions (like ftello) are
only defined at a higher POSIX level.
Why is that a problem?
Because Python tries to use them.
Post by Martin v. Loewis
Post by Marc Recht
In that case __BSD_VISIBLE is not set.
And why is that a problem?
Because certain stuff Python seems to rely on isn't there. (see
SourceForge)

An other option could be that Python doesn't use non-POSIX
functions/definitions were they aren't available..
Post by Martin v. Loewis
This is exhausting.
I know and talking to both camps is rather frustrating..

Maybe we should continue this discussion on SourceForge.

Regards,
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-11 19:56:52 UTC
Permalink
Post by Marc Recht
Sorry. Trying to. I've posted more detail (including a build log) on
SourceForge. But again the major problem is certain defines and
typedefs like PF_INET or u_char, u_int (see SourceForge) are only
defined in the __BSD_VISIBLE case. These defines typedefs are needed
by certain Python modules like sockemodule, nismodule.
As I commented on SF: Python does not use u_char or u_int. So if that
fails to compile, it's a bug on FreeBSD (it is the system headers,
after all, which fail to compile - something I have no regrets for).
Post by Marc Recht
I also a friend of generic/standard solutions,I don't see one here. I
would be more than happy proved to be wrong...
Taking the failure to compile system headers away (which really needs
to be fixed in the system), then Python does compile correctly,
doesn't it? Looks like an almost-generic solution to me (if it wasn't
for the warnings about implicitly-defined functions - which all get
defined correctly implicitly).
Post by Marc Recht
"The whole point of the standards constants is to specify a strict
environment. If you want a BSD environment don't specify a particular
standard, it's simple.".
We do want a POSIX environment. If the system offers alternatives, we
want the POSIX alternative, not the native alternative. BSD has a
tradition of doing things differently than POSIX, so I have the strong
urge to tell the system "I want POSIX, dammit". If the system offers
extensions over POSIX, we want them as well. There is unfortunately no
standard way to access those extensions, but most systems do have a
means to access them (while *still* allowing to favour POSIX over any
native semantics). It is really unfortunate that neither OpenBSD nor
FreeBSD offer such an environment.
Post by Marc Recht
I understand that, but this is a build fix. So the evaluation is quite
simple..
No, it isn't. Python builds just fine without your patch, if we
consider the bugs in the system headers fixed.
Post by Marc Recht
Post by Martin v. Loewis
Post by Marc Recht
Some problems are left, because some functions (like ftello) are
only defined at a higher POSIX level.
Why is that a problem?
Because Python tries to use them.
Ah, that is indeed a problem. Autoconf is bad in finding out that a
function is available, but has no prototype. This can be fixed (on a
case-by-case basis), and I have fixed it for chroot, link, and
symlink.

This is not a big problem, though. In C, a prototype is implied from
the first call, and that implied prototype is correct.
Post by Marc Recht
Because certain stuff Python seems to rely on isn't there. (see
SourceForge)
Python relies on none of this. Instead, Python wraps the functions,
and exposes them to the application. There is no "internal" use of any
of the features we are talking about. So if configure determines that
a feature is absent, it just won't be exposed. No big deal.
Post by Marc Recht
An other option could be that Python doesn't use non-POSIX
functions/definitions were they aren't available..
That is indeed an option. As I said, Python does *not* use these
functions. Not wrapping them is really simple.

Regards,
Martin

Marc Recht
2002-11-10 21:12:40 UTC
Permalink
Post by Hye-Shik Chang
But, this build gets segfault on installing.
... make install ...
./python -E ./setup.py install --prefix=/home/perky/test --install-scripts=/home/perky/test/bin --install-platlib=/home/perky/test/lib/python2.3/lib-dynload
running install
running build
running build_ext
Segmentation fault (core dumped)
*** Error code 139
Try with "--without-pymalloc" that solved this issue for me. But the
tests still fail in "test_re".

Marc
Guido van Rossum
2002-11-11 00:52:52 UTC
Permalink
Post by Marc Recht
Try with "--without-pymalloc" that solved this issue for me. But the
tests still fail in "test_re".
Marc, pymalloc is supposed to be bulletproof. If there's a segfault
that can be avoided by disabling pymalloc, that's a bug in pymalloc.
Would you mind helping us find this bug?

--Guido van Rossum (home page: http://www.python.org/~guido/)
Tim Peters
2002-11-11 01:28:24 UTC
Permalink
[Guido]
Post by Guido van Rossum
Marc, pymalloc is supposed to be bulletproof.
Not necessarily. It's not possible to write a memory manager in 100% std C,
and some platform may be bizarre enough that pymalloc just can't be used on
it. We haven't found one yet, though. Simple example: if some platform
requires 16-byte aligned addresses, pymalloc will plain fail on that
platform. Or, I'm unsure what would happen on a word-adressed machine, but
I am sure it wouldn't be pretty <wink>. Or the OS and/or HW may do read
protection at the byte level, so that pymalloc's belief that it can always
read stuff from "the next lower pool boundary" is wrong here. There are
lots of things that *could* go wrong.
Post by Guido van Rossum
If there's a segfault that can be avoided by disabling pymalloc, that's
a bug in pymalloc.
Again not necessarily: there's tricky casting code throughout pymalloc (as
there is in any low-level memory manager), and a compiler optimization
and/or codegen bug would be my first guess. So the first thing I'd do is
recompile pymalloc without optimization.
Post by Guido van Rossum
Would you mind helping us find this bug?
Please yes -- there's no way to guess.
Marc Recht
2002-11-11 10:00:33 UTC
Permalink
Post by Guido van Rossum
Marc, pymalloc is supposed to be bulletproof. If there's a segfault
that can be avoided by disabling pymalloc, that's a bug in pymalloc.
Would you mind helping us find this bug?
No problem. Environment: FreeBSD-current (Nov. 8th), Python CVS Sources
Nov. 11th (~ 10:00 CEST).
limit:
cputime unlimited
filesize unlimited
datasize 512MB
stacksize 64MB
coredumpsize unlimited
memoryuse unlimited
memorylocked unlimited
maxproc 7390
descriptors 32768
sockbufsize unlimited
vmemorysize unlimited

I compiled Python with no optimization flags set and my changes to
pyconfig.h.in/configure.in (removes XOPEN_*,POSIX_* in the FreeBSD case
to get it compiled). The only configure argument was prefix.

Error
/usr/bin/install -c ./install-sh
/opt/local/python/lib/python2.3/config/install-sh
./python -E ./setup.py install \
--prefix=/opt/local/python \
--install-scripts=/opt/local/python/bin \
--install-platlib=/opt/local/python/lib/python2.3/lib-dynload
running install
running build
running build_ext
gmake: *** [sharedinstall] Segmentation fault (core dumped)

713 if (ADDRESS_IN_RANGE(p, pool->arenaindex)) {

(gdb) bt
#0 0x080779c0 in PyObject_Free (p=0x800) at Objects/obmalloc.c:713
#1 0x080e00a0 in function_call (func=0x82641ec, arg=0x8258b8c,
kw=0x826bbdc) at Objects/funcobject.c:481
#2 0x080599fb in PyObject_Call (func=0x82641ec, arg=0x8258b8c,
kw=0x826bbdc) at Objects/abstract.c:1688
#3 0x080a7950 in ext_do_call (func=0x82641ec, pp_stack=0xbfbfdfa4,
flags=2, na=1, nk=0) at Python/ceval.c:3453
#4 0x080a4b64 in eval_frame (f=0x81d400c) at Python/ceval.c:2043
#5 0x080a5e1e in PyEval_EvalCodeEx (co=0x820e620, globals=0x819a57c,
locals=0x0, args=0x831ad70, argcount=1, kws=0x831ad74, kwcount=0,
defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:2554
#6 0x080a73c3 in fast_function (func=0x82378ec, pp_stack=0xbfbfe194,
n=1, na=1, nk=0) at Python/ceval.c:3297
#7 0x080a72af in call_function (pp_stack=0xbfbfe194, oparg=0) at
Python/ceval.c:3266
#8 0x080a4a50 in eval_frame (f=0x831ac0c) at Python/ceval.c:2009
[...]

(gdb) l
708
709 if (p == NULL) /* free(NULL) has no effect */
710 return;
711
712 pool = POOL_ADDR(p);
713 if (ADDRESS_IN_RANGE(p, pool->arenaindex)) {
714 /* We allocated this address. */
715 LOCK();
716 /*
717 * Link p to the start of the pool's freeblock list. Since

Here ?

(gdb) p pool
$1 = (struct pool_header *) 0x0

(gdb) p p
$2 = (void *) 0x800

(gdb) x 0x800
0x800: Cannot access memory at address 0x800

This doesn't happen if either --without-pymalloc or --with-pydebug is
given.

HTH,
Marc
--
"Premature optimization is the root of all evil." -- Donald E. Knuth
Martin v. Loewis
2002-11-11 12:43:57 UTC
Permalink
Post by Marc Recht
#0 0x080779c0 in PyObject_Free (p=0x800) at Objects/obmalloc.c:713
#1 0x080e00a0 in function_call (func=0x82641ec, arg=0x8258b8c,
kw=0x826bbdc) at Objects/funcobject.c:481
This looks like it is getting difficult. 0x800 is surely a garbage
pointer, so it is not surprising that pymalloc crashes when it sees
that pointer.

Now, the question is: where does that pointer come from? This should
be from funcobject.c:457, which says

k = PyMem_NEW(PyObject *, 2*nk);

Unless I'm mistaken, this expands to malloc(). Can you please try to
confirm that malloc indeed return this 0x800? If so, it looks like the
C library's malloc got confused. We would then need to find out why
that happens.

If you can't come up with a debugging strategy, I recommend that you
try out Tim's suggestion (ie. build with optimizations disabled). In
addition, I also suggest that you build with threads disabled. Of
course, if you then cannot reproduce the problem anymore, this is
little proof that the problem has to do with optimization, or threads
:-(

Regards,
Martin
Tim Peters
2002-11-11 15:47:55 UTC
Permalink
[Martin v. Loewis]
Post by Martin v. Loewis
...
Now, the question is: where does that pointer come from? This should
be from funcobject.c:457, which says
k = PyMem_NEW(PyObject *, 2*nk);
Unless I'm mistaken, this expands to malloc().
Which version of Python is getting built here?

In 2.3, PyMem_NEW resolves to malloc() in a release build, but not a debug
build (in a debug build, all Python memory API calls go through pymalloc).
This may be vaguely <wink> relevant, since Marc said

This doesn't happen if either --without-pymalloc or --with-pydebug
is given.

In the former case (--without-pymalloc) pymalloc isn't used at all; in the
latter case (--with-pydebug) pymalloc is always used. In 2.3. Since I saw
a

/opt/local/python/lib/python2.3/config/install-sh
^^^^^^^^^

flash by, I figure that's right for this build.
Martin v. Loewis
2002-11-11 18:22:30 UTC
Permalink
Post by Tim Peters
In 2.3, PyMem_NEW resolves to malloc() in a release build, but not a debug
build (in a debug build, all Python memory API calls go through pymalloc).
This may be vaguely <wink> relevant, since Marc said
This doesn't happen if either --without-pymalloc or --with-pydebug
is given.
To summarize your analysis: In Marc's build, malloc is being called
directly, for PyMem_NEW. Right?

Regards,
Martin
Tim Peters
2002-11-11 19:29:51 UTC
Permalink
Post by Martin v. Loewis
To summarize your analysis: In Marc's build, malloc is being called
directly, for PyMem_NEW. Right?
I haven't read a Unixish build trail in years. If he was doing a 2.3
release build (my best guess), yes.
Loading...