New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
shared array eating memory #5802
Comments
From @lizmatThe following program grows to more than 250 Mbyte of RAM within Seems to me we have a (huge) leak here. ;-( ================================================== my $thread = threads->new( push( @queue,1 ) while 1;For simplicity, I did not use any lock()s, as shift() and push() Perl Info
|
From @lizmatAt 12:00 PM 7/31/02 +0000, via RT wrote:
The same happens with: use threads; my $thread = threads->new( my $i; so the problem does not seem related to shared arrays, but also applies to Liz |
From nick.ing-simmons@elixent.comElizabeth Mattijsen <perl5-porters@perl.org> writes:
What says the shift-ing thread has to be given any CPU time, Also once the thread does die - remember you won't see the message
|
From @lizmatAt 08:37 AM 8/1/02 +0100, Nick Ing-Simmons wrote:
So you're saying the thread may never see that the array has grown too
Well, that's just the point. It _never_ dies (at least not before taking So something is not being freed or re-used or whatever. Maybe technically Liz |
From goldbb2@earthlink.netNick Ing-Simmons wrote:
If you're worried that the thread is dieing and not ever being given the use threads; -- |
From @lizmatAt 05:04 AM 8/1/02 -0400, Benjamin Goldberg wrote:
Eh... I'm not interested in whether this works or not. The point is that The point is that I'm trying to show that there is a _HUGE_ leak with Liz |
From @iabynOn Thu, Aug 01, 2002 at 11:07:39AM +0200, Elizabeth Mattijsen wrote:
But as Nick has pointed out, your code doesn't demonstrate this. For one possibility of how the threads get scheduled, your program is push @array, 1 while 1; Which is not to say there may not be leaks, but your particular example -- |
From @lizmatAt 11:19 AM 8/1/02 +0100, Dave Mitchell wrote:
Except that the elements are removed again within the thread as fast as the There was some concern that the thread that is removing elements from the use threads;
Have you looked at "top" when this program runs? I'd say that is enough Anyway, I was pointed to this particular problem by a user on the Needless to say this user is not using threads anymore, although the memory Liz |
From @iabynOn Thu, Aug 01, 2002 at 12:39:01PM +0200, Elizabeth Mattijsen wrote:
Okay, I've tried it with some modified code that I'm satified works, and Sounds like a job for Arthur... Dave. use threads; my @queue : shared; -- |
From @lizmatAt 12:22 PM 8/1/02 +0100, Dave Mitchell wrote:
No problem... Thanks for checking it out and confirming what I saw... I hope I wasn't too pushy... I was getting "parrot sketch"
Indeed... ;-) I hope it is going to be a simple patch that can be applied Liz |
From arthur@contiller.seOn torsdag, augusti 1, 2002, at 01:49 , Elizabeth Mattijsen wrote:
Grr, I will take a look at it this weekend, hopefully it it is not a Arthur |
From @lizmatAt 02:48 PM 8/1/02 +0200, Arthur Bergman wrote:
That would even be better! Liz |
From nick.ing-simmons@elixent.comElizabeth Mattijsen <liz@dijkmat.nl> writes:
But there is nothing in the code that forces that, it is just an artifact
The thread dies, the process doesn't.
There is no gate in the main thread doing the pushing, so it just keeps #!perl
In a production environment the "producer" should be the one to test the e.g. while (1) {
|
From nick.ing-simmons@elixent.comElizabeth Mattijsen <liz@dijkmat.nl> writes:
You have written the equivalent of: perl -e 'push(@queue,1) while 1' and it behaves the same: nick@bactrian 1030$ time perl -e 'push(@queue,1) while 1' real 0m27.027s
No you haven't - you have just re-discovered that infinite loops with
|
From nick.ing-simmons@elixent.comElizabeth Mattijsen <liz@dijkmat.nl> writes:
No. The pulling off thread is going to be slower 'cos it does the
There is no "die" in the main thread, nor a join to propagate the die
And I am afraid to say you have only exposed your threads programming skills. use threads; while (1) { -- |
From @lizmatAt 03:15 PM 8/1/02 +0100, Nick Ing-Simmons wrote:
And take any other thread that's running with it. Because that's the way
Eh... in Perl, a C<die> in a thread is propagated to all other threads
Well, I'm eager to learn and not afraid to get the lid on my nose... How
Well, on my development box, this eats memory just as fast as my original A die in one thread is the same as a die in any other thread. So in Perl I guess Perl threads programming in that respect is more like real Liz |
From cmeyer@helvella.orgOn Wed, Jul 31, 2002 at 12:00:23PM -0000, Elizabeth Mattijsen wrote:
perldoc perlthrtut says: Even "$a += 5" or "$a++" are not guaranteed to be atomic. shift() and push() are a bit more complex than increment, so how am I to perlfunc, threads::shared, perlguts, perlapi and the like don't seem to Oh, now I see that threads::shared uses a tied interface to do its stuff Perhaps the threads::shared pod should talk about what's atomic (and internals newbie, thinking aloud |
From @lizmatAt 08:45 PM 8/1/02 -0700, Colin Meyer wrote:
Just before the release of 5.8.0 there was a discussion as to whether the However, since then, Arthur and Benjamin have confirmed that: - shift(), unshift(), push() and pop() are self-locking, I would rather not call these atomic (they are), but they don't describe
Does that matter? Maybe in a pod describing internals? Liz |
From cmeyer@helvella.orgOn Fri, Aug 02, 2002 at 08:30:25AM +0200, Elizabeth Mattijsen wrote:
Fair enough: the programmer using shared variables should not assume
Yes, it does matter that sharing a hash or an array ties it. Try tying -Colin. |
From @lizmatAt 11:55 PM 8/1/02 -0700, Colin Meyer wrote:
Ah, I think that that is actually classified as a bug, actually, to be Liz |
From nick.ing-simmons@elixent.comElizabeth Mattijsen <liz@dijkmat.nl> writes:
That isn't the way I coded it, and as far as I can "die" only kills the thread that calls it. This is because has an implied XS level eval {} round the call. which catches the die. Once upon a time the $@ value was propagated to whoever called ->join
Which is why I was not really worried that the "obvious" way I coded
Er, I wrote the code and I am reasonably sure it doesn't work like that.
I think this is our fundamental difference of opinion. Perhaps Arthur Meanwhile I will build a fresh threaded perl and experiment some. -- |
From nick.ing-simmons@elixent.comColin Meyer <cmeyer@helvella.org> writes:
Scalars are shared with some 'magic' as well - but not tie magic, -- |
From @lizmatAt 10:01 AM 8/2/02 +0100, Nick Ing-Simmons wrote:
Hmmm... a simple example proves that you're right. And that I'm
Yes, you're right. Still, the die() would have put a message on the
Yes, indeed you're completely right and I'm completely wrong. Again, the
Not anymore...
Now, that is a good idea... ;-) And thanks for putting up with me in Liz |
From arthur@contiller.seOn fredag, augusti 2, 2002, at 05:45 , Colin Meyer wrote:
Each statement is atomic, $a++ is not atomic since it is the same as
Common sense, they only access the variable once. Arthur |
From arthur@contiller.seOn fredag, augusti 2, 2002, at 08:55 , Colin Meyer wrote:
That would seem to be a very suicidal thing to do anyway :-) Arthur |
From @rgarciaArthur Bergman wrote:
There's a separate opcode for $a++ (op_preinc), so it's not the same
|
From arthur@contiller.seOn fredag, augusti 2, 2002, at 01:06 , Rafael Garcia-Suarez wrote:
shhhh ;), it still access the variable twice Arthur |
From @lizmatAt 01:06 PM 8/2/02 +0200, Rafael Garcia-Suarez wrote:
There was a thread about this about 6 weeks ago. Basically try having 10 So, ++ is not atomic... Liz |
From @TuxOn Fri 02 Aug 2002 13:06, Rafael Garcia-Suarez <raphel.garcia-suarez@hexaflux.com> wrote:
Hmm, I hope I'm not wrong, but IIRC that is for ++$a, and not for $a++ ++$a is atomic, $a++ is not
-- |
From nick.ing-simmons@elixent.comArthur Bergman <arthur@contiller.se> writes:
I find that even after 20 years "common sense" and "concurent programming"
|
From arthur@contiller.seOn fredag, augusti 2, 2002, at 02:11 , H.Merijn Brand wrote:
Nope, none of them are atomic, they could be made atomic by locking the Patches welcome if they don't slow down non shared variables! Arthur |
From cmeyer@helvella.orgOn Fri, Aug 02, 2002 at 01:05:07PM +0200, Arthur Bergman wrote:
Hrm. I had hoped that with BerkeleyDB.pm and proper locking that However, what really concerns me is that one cannot share any tied hash, -Colin. |
From jim.cistaro@av.comI am a relative novice as XS so please pardon my ignorance. I Is there a reason that POP and unshift use Also should they do a I have only done minimal testing of a simple queue, but this seems to As I said, I am somewhat of a novice at XS and just want to be able to Jim |
arthur@contiller.se - Status changed from 'new' to 'open' |
From arthur@contiller.seBug fixed, All shared variables were created with a reference number of 2. Patch fixes this, in the repository as #19200. Arthur |
From arthur@contiller.seInline Patch--- shared.xs-19199 Sun Apr 13 21:54:30 2003
+++ shared.xs Sun Apr 13 21:54:32 2003
@@ -275,8 +275,10 @@
/* If neither of those then create a new one */
if (!data) {
SHARED_CONTEXT;
- if (!ssv)
+ if (!ssv) {
ssv = newSV(0);
+ SvREFCNT(ssv) = 0;
+ }
data = PerlMemShared_malloc(sizeof(shared_sv));
Zero(data,1,shared_sv);
SHAREDSvPTR(data) = ssv;
@@ -503,7 +505,6 @@
assert ( SHAREDSvPTR(shared) );
ENTER_LOCK;
-
if (SvTYPE(SHAREDSvPTR(shared)) == SVt_PVAV) {
assert ( mg->mg_ptr == 0 );
SHARED_CONTEXT;
@@ -782,6 +783,7 @@
sharedsv_scalar_store(aTHX_ tmp, target);
SHARED_CONTEXT;
av_push((AV*) SHAREDSvPTR(shared), SHAREDSvPTR(target));
+ SvREFCNT_inc(SHAREDSvPTR(target));
SHARED_RELEASE;
SvREFCNT_dec(tmp);
}
@@ -801,6 +803,7 @@
sharedsv_scalar_store(aTHX_ tmp, target);
SHARED_CONTEXT;
av_store((AV*) SHAREDSvPTR(shared), i - 1, SHAREDSvPTR(target));
+ SvREFCNT_inc(SHAREDSvPTR(target));
CALLER_CONTEXT;
SvREFCNT_dec(tmp);
} |
arthur@contiller.se - Status changed from 'open' to 'resolved' |
From guest@guest.guest.xxxxxxxx[sky - Sun Apr 13 13:03:29 2003]:
Is there someplace where can I get the compiled binaries for this fix |
From @lizmatAt 05:34 +0000 5/22/03, Guest (via RT) wrote:
I'm not aware of any such place. Liz |
From guest@guest.guest.xxxxxxxxI did a local build using the patch in this thread for the shared.xs. "panic: COND_INIT (1816)." You can use the following test porgrams to reproduce the problem. I Details of my perl (part of perl -V o/p): #-------test program 1----------# use strict; use threads; my @a:shared = (1,2,3); sub f1 { sub f2 { undef (@a); if ($#wq >= 0) { my $thr = threads->new (\&f2); #-------test program 2----------# #! /usr/local/bin/perl -w use strict; use threads; my @a:shared = (1,2,3); sub f1 { sub f2 { undef (@a); if ($#wq >= 0) { my $thr = threads->new (\&f2); Any insight is really appreciated! Thanks, [elizabeth - Thu May 22 09:52:07 2003]:
|
From arthur@contiller.seI think that the test cases are flawed, for example, the yield is pointless since you still However, I did run them and saw no memory leak, however as far as I can tell the array please try with a snapshot of maintperl or bleadperl and see if you can reproduce the [guest - Mon May 26 22:51:53 2003]:
|
arthur@contiller.se - Status changed from 'open' to 'resolved' |
From guest@guest.guest.xxxxxxxx[sky - Mon Jun 9 06:57:37 2003]:
Now, if you consider the above sequence to repeat, it would so happen
If you look at the code, the shared array is being emptied by the
Thanks, |
From g@netcraft.com.auOn Mon, Jun 09, 2003 at 01:57:38PM -0000, Arthur Bergman wrote:
Here is a much simpler test case: #!/usr/bin/perl -w use strict; use threads; my $q = Thread::Queue->new(1); $q->enqueue(1) while $q->dequeue(); The patch didn't seem to fix the problem for me, but maybe I did -- |
From arthur@contiller.se[gdb - Tue Jun 10 04:29:18 2003]:
What platform are you running on, we can reproduce this on tru64 but not any other. Arthur |
From g@netcraft.com.auOn Thu, Jun 12, 2003 at 11:00:59AM -0000, Arthur Bergman wrote:
Red Hat Linux 9 Intel: $ uname -a -- |
From mterretta@advection.netHave a 2,000 line program on ActivePerl Build 805 for Win32 that processes Started whittling away the code until I had only an enqueue and a dequeue Found the following even simpler program, given on Google to illustrate the use strict; my $q = Thread::Queue->new(1); $q->enqueue(1) while $q->dequeue(); It was noted this only reproduces on Tru64. Well, it reproduces on Win32 as My perl -V here: Summary of my perl5 (revision 5 version 8 subversion 0) configuration: Characteristics of this binary (from libperl): |
@cpansprout - Status changed from 'open' to 'resolved' |
Migrated from rt.perl.org#15893 (status was 'resolved')
Searchable as RT15893$
The text was updated successfully, but these errors were encountered: