Neko thread limit

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Neko thread limit

Max S-3
Hi.

It seems that there is no function for stopping threads in
neko.vm.Thread and the thread limit of 300 (on my system) is filled
very quickly if I spawn threads for some tasks. Codewise the threads
are dead when their callables finish working but it's not so
system-wise.

The following code dies in seconds, though thread callable does
nothing at all and thread should be detached upon finishing.

import neko.vm.Thread;

class Server
{
  static var s:Server;

  static function main()
    {
      for (i in 1...1000)
        Thread.create(runThread);
    }


  public static function runThread()
    {
      trace("run");
    }
}

Is there a proper way to use threads in neko?

Thanks.

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Nicolas Cannasse
Max S a écrit :
> Hi.
>
> It seems that there is no function for stopping threads in
> neko.vm.Thread and the thread limit of 300 (on my system) is filled
> very quickly if I spawn threads for some tasks. Codewise the threads
> are dead when their callables finish working but it's not so
> system-wise.
[...]
> Is there a proper way to use threads in neko?

Thread creation is an expensive operation anyway and a thread takes
quite a lot of memory, so you shouldn't be using threads as task runners.

One possibility is to use a shared queue on which several threads can
wait for events to execute :

function wait() {
        while( true ) {
                var f = queue.pop(true);
                f();
        }
}

queue = new neko.vm.Dequeue();
for( i in 0...32 )
        neko.vm.Thread.create(wait);

then you can add events into the queue :

queue.add(function() trace("Hello"));

Nicolas


--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Max S-3
Thank you for the solution. But what about the thread limit? If any
application spawns threads (even not that often) it will eventually
hit the limit without detach function available. That means dynamic
thread creation is pretty much out of the question. Isn't that a bug?

2009/7/6 Nicolas Cannasse <[hidden email]>:

> Max S a écrit :
>>
>> Hi.
>>
>> It seems that there is no function for stopping threads in
>> neko.vm.Thread and the thread limit of 300 (on my system) is filled
>> very quickly if I spawn threads for some tasks. Codewise the threads
>> are dead when their callables finish working but it's not so
>> system-wise.
>
> [...]
>>
>> Is there a proper way to use threads in neko?
>
> Thread creation is an expensive operation anyway and a thread takes quite a
> lot of memory, so you shouldn't be using threads as task runners.
>
> One possibility is to use a shared queue on which several threads can wait
> for events to execute :
>
> function wait() {
>        while( true ) {
>                var f = queue.pop(true);
>                f();
>        }
> }
>
> queue = new neko.vm.Dequeue();
> for( i in 0...32 )
>        neko.vm.Thread.create(wait);
>
> then you can add events into the queue :
>
> queue.add(function() trace("Hello"));
>
> Nicolas
>
>
> --
> haXe - an open source web programming language
> http://haxe.org
>

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Nicolas Cannasse
Max S a écrit :
> Thank you for the solution. But what about the thread limit? If any
> application spawns threads (even not that often) it will eventually
> hit the limit without detach function available. That means dynamic
> thread creation is pretty much out of the question. Isn't that a bug?

Threads should be reclaimed, on which OS are you experimenting leaking ?

Nicolas

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

blackdog-2
I've been doing some neko threading on linux 2.26 and if I just spawn a
thread to be executed once, e.g. I only read one message sent from the
main thread ...

worker = neko.vm.Thread.create(myworker);
worker.sendMessage(myfunc);

.
.
.

function myworker() {
var f = neko.vm.Thread.readMessage(true);
f();
}

the memory, at least, is not reclaimed according to gnome system
monitor, i get a continual increase making thread on demand (rather than
pooled) unacceptable.


bd

On Mon, 2009-07-06 at 20:55 +0200, Nicolas Cannasse wrote:

> Max S a écrit :
> > Thank you for the solution. But what about the thread limit? If any
> > application spawns threads (even not that often) it will eventually
> > hit the limit without detach function available. That means dynamic
> > thread creation is pretty much out of the question. Isn't that a bug?
>
> Threads should be reclaimed, on which OS are you experimenting leaking ?
>
> Nicolas
>


--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Nicolas Cannasse
ritchie turner a écrit :

> I've been doing some neko threading on linux 2.26 and if I just spawn a
> thread to be executed once, e.g. I only read one message sent from the
> main thread ...
>
> worker = neko.vm.Thread.create(myworker);
> worker.sendMessage(myfunc);
>
> .
> .
> .
>
> function myworker() {
> var f = neko.vm.Thread.readMessage(true);
> f();
> }
>
> the memory, at least, is not reclaimed according to gnome system
> monitor, i get a continual increase making thread on demand (rather than
> pooled) unacceptable.

There are two things to consider here :

a) there was a fix on NekoVM CVS related to pthreads which might prevent
created threads to leak

b) the GC will not always free the messages immediately. Depending on
your application, it will consume memory until it reach a stable level
at which allocation and GC cycles are balanced

Nicolas


--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

blackdog-2

ok, thanks.

bd

On Mon, 2009-07-06 at 22:57 +0200, Nicolas Cannasse wrote:

> ritchie turner a écrit :
> > I've been doing some neko threading on linux 2.26 and if I just spawn a
> > thread to be executed once, e.g. I only read one message sent from the
> > main thread ...
> >
> > worker = neko.vm.Thread.create(myworker);
> > worker.sendMessage(myfunc);
> >
> > .
> > .
> > .
> >
> > function myworker() {
> > var f = neko.vm.Thread.readMessage(true);
> > f();
> > }
> >
> > the memory, at least, is not reclaimed according to gnome system
> > monitor, i get a continual increase making thread on demand (rather than
> > pooled) unacceptable.
>
> There are two things to consider here :
>
> a) there was a fix on NekoVM CVS related to pthreads which might prevent
> created threads to leak
>
> b) the GC will not always free the messages immediately. Depending on
> your application, it will consume memory until it reach a stable level
> at which allocation and GC cycles are balanced
>
> Nicolas
>
>


--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Max S-3
In reply to this post by Nicolas Cannasse
On Linux (Fedora 9). Neko is default 1.8.0.

2009/7/6 Nicolas Cannasse <[hidden email]>:

> Max S a écrit :
>>
>> Thank you for the solution. But what about the thread limit? If any
>> application spawns threads (even not that often) it will eventually
>> hit the limit without detach function available. That means dynamic
>> thread creation is pretty much out of the question. Isn't that a bug?
>
> Threads should be reclaimed, on which OS are you experimenting leaking ?
>
> Nicolas
>
> --
> haXe - an open source web programming language
> http://haxe.org
>

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Nicolas Cannasse
Max S a écrit :
> On Linux (Fedora 9). Neko is default 1.8.0.

You can maybe try with 1.8.1 CVS.

But your example will maybe not work since the main thread might create
too much of them before they are able to finish.

Nicolas

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Max S-3
Running on version from CVS gives the same result (Though the version
number still shows 1.8.0, no idea if I pulled the right one). It maybe
is true that thread dies only after a certain timeout, but why haven't
you made the explicit pthread_detach() call available for use in haxe?
What if some running thread needs to be killed from the another for
example if it takes too long?

2009/7/7 Nicolas Cannasse <[hidden email]>:

> Max S a écrit :
>>
>> On Linux (Fedora 9). Neko is default 1.8.0.
>
> You can maybe try with 1.8.1 CVS.
>
> But your example will maybe not work since the main thread might create too
> much of them before they are able to finish.
>
> Nicolas
>
> --
> haXe - an open source web programming language
> http://haxe.org
>

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Nicolas Cannasse
Max S a écrit :
> Running on version from CVS gives the same result (Though the version
> number still shows 1.8.0, no idea if I pulled the right one). It maybe
> is true that thread dies only after a certain timeout, but why haven't
> you made the explicit pthread_detach() call available for use in haxe?

Because it's not crossplatform (there's no detach in Windows Threads)

But in 1.8.0/CVS (will be 1.8.1 when released) the threads are created
in detached mode.

> What if some running thread needs to be killed from the another for
> example if it takes too long?

You can't kill a thread in Neko.

Nicolas

--
haXe - an open source web programming language
http://haxe.org
Reply | Threaded
Open this post in threaded view
|

Re: Neko thread limit

Zenko Klapko Jr.
Max: I ran into the same problem on linux as well and made some posts to
the irc chat regarding it. I do consider a bug, design flaws in my code aside.

Nicholas: Windows does have a pthreads implementation, I've used it
before for creating cross-platform threaded applications.
http://sourceware.org/pthreads-win32/

To program around the 'out to lunch' thread, a timer within the thread
can determine if something's wrong and return. This feels reminscent of
programming interrupts...

-Zenko

On Tue, Jul 07, 2009 at 09:05:14PM +0200, Nicolas Cannasse wrote:

> Max S a écrit :
> >Running on version from CVS gives the same result (Though the version
> >number still shows 1.8.0, no idea if I pulled the right one). It maybe
> >is true that thread dies only after a certain timeout, but why haven't
> >you made the explicit pthread_detach() call available for use in haxe?
>
> Because it's not crossplatform (there's no detach in Windows Threads)
>
> But in 1.8.0/CVS (will be 1.8.1 when released) the threads are created
> in detached mode.
>
> >What if some running thread needs to be killed from the another for
> >example if it takes too long?
>
> You can't kill a thread in Neko.
>
> Nicolas
>
> --
> haXe - an open source web programming language
> http://haxe.org
--
I would rather be exposed to the inconveniences attending too much liberty than to those attending too small a degree of it.
 - Thomas Jefferson

--
haXe - an open source web programming language
http://haxe.org

signature.asc (196 bytes) Download Attachment