Re: script uses up all memory topic

On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell <(E-Mail Removed)> wrote:
> I have a script that forks off other processes and attempts to manage
> them. Here is a stripped down version of the script:
>
> self.sleepTime = 300
> self.procs = {}
> self.startTimes = {}
> self.cmd = ['python', '/usr/local/motor/motor/app/some_other_script.py']
>
> while True:
> try:
> self.tools = Tool.objects.filter(ip__isnull=False)
> except Exception, e:
> print 'error from django call: ' + str(e)
> sys.exit(1)
>
> for tool in self.tools:
> name = tool.name
> if name in self.procs:
> if self.procs[name].poll() is None:
> if
> (datetime.datetime.now()-self.startTimes[name]) >
> datetime.timedelta(hours=12):
> # it's been running too long - kill it
> print 'killing script for ' + name + "
> it's been running too long"
> self.procs[name].kill()
> else:
> continue
>
> if self.procs[name].returncode:
> print 'scrikpt failed for ' + name + ', error
> = ' + str(self.procs[name].returncode)
>
> print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
> try:
> self.procs[name] = subprocess.Popen(self.cmd)
> self.startTimes[name] = datetime.datetime.now()
> except Exception, e:
> print 'error from Popen: ' + str(e)
> sys.exit(1)
> else:
> print 'starting script.py for ' + name + ' at ' +
> str(datetime.datetime.now())
> try:
> self.procs[name] = subprocess.Popen(self.cmd)
> self.startTimes[name] = datetime.datetime.now()
> except Exception, e:
> print 'error from Popen: ' + str(e)
> sys.exit(1)
>
> time.sleep(self.sleepTime)
>
>
> The script does what it's intended to do, however after about 2 hours
> it has used up all the memory available and the machine hangs. Can
> anyone see something that I am doing here that would be using memory
> like this? Perhaps some immutable object needs to be repeatedly
> recreated?

I figured out what is causing this. Each pass through the loop it does:

self.tools = Tool.objects.filter(ip__isnull=False)

And that is what is causing the memory consumption. If I move that
outside the loop and just do that once the memory issue goes away. Now
I need to figure out why this is happening and how to prevent it as
they do want to query the db each pass through the loop in case it has
been updated.

Re: script uses up all memory topic

On Thu, Mar 6, 2014 at 4:56 PM, Larry Martell <(E-Mail Removed)> wrote:
> On Wed, Mar 5, 2014 at 5:27 PM, Larry Martell <(E-Mail Removed)> wrote:
>> I have a script that forks off other processes and attempts to manage
>> them. Here is a stripped down version of the script:
>>
>> self.sleepTime = 300
>> self.procs = {}
>> self.startTimes = {}
>> self.cmd = ['python', '/usr/local/motor/motor/app/some_other_script.py']
>>
>> while True:
>> try:
>> self.tools = Tool.objects.filter(ip__isnull=False)
>> except Exception, e:
>> print 'error from django call: ' + str(e)
>> sys.exit(1)
>>
>> for tool in self.tools:
>> name = tool.name
>> if name in self.procs:
>> if self.procs[name].poll() is None:
>> if
>> (datetime.datetime.now()-self.startTimes[name]) >
>> datetime.timedelta(hours=12):
>> # it's been running too long - kill it
>> print 'killing script for ' + name + "
>> it's been running too long"
>> self.procs[name].kill()
>> else:
>> continue
>>
>> if self.procs[name].returncode:
>> print 'scrikpt failed for ' + name + ', error
>> = ' + str(self.procs[name].returncode)
>>
>> print 'starting script.py for ' + name + ' at ' +
>> str(datetime.datetime.now())
>> try:
>> self.procs[name] = subprocess.Popen(self.cmd)
>> self.startTimes[name] = datetime.datetime.now()
>> except Exception, e:
>> print 'error from Popen: ' + str(e)
>> sys.exit(1)
>> else:
>> print 'starting script.py for ' + name + ' at ' +
>> str(datetime.datetime.now())
>> try:
>> self.procs[name] = subprocess.Popen(self.cmd)
>> self.startTimes[name] = datetime.datetime.now()
>> except Exception, e:
>> print 'error from Popen: ' + str(e)
>> sys.exit(1)
>>
>> time.sleep(self.sleepTime)
>>
>>
>> The script does what it's intended to do, however after about 2 hours
>> it has used up all the memory available and the machine hangs. Can
>> anyone see something that I am doing here that would be using memory
>> like this? Perhaps some immutable object needs to be repeatedly
>> recreated?
>
> I figured out what is causing this. Each pass through the loop it does:
>
> self.tools = Tool.objects.filter(ip__isnull=False)
>
> And that is what is causing the memory consumption. If I move that
> outside the loop and just do that once the memory issue goes away. Now
> I need to figure out why this is happening and how to prevent it as
> they do want to query the db each pass through the loop in case it has
> been updated.

Apparently the object returned by that call is immutable as if I look
at id(self.tools) each pass through the loop, it is different. Is
there some way I can recover that memory?

Re: script uses up all memory topic

On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell <(E-Mail Removed)> wrote:
> I figured out what is causing this. Each pass through the loop it does:
>
> self.tools = Tool.objects.filter(ip__isnull=False)
>
> And that is what is causing the memory consumption. If I move that
> outside the loop and just do that once the memory issue goes away. Now
> I need to figure out why this is happening and how to prevent it as
> they do want to query the db each pass through the loop in case it has
> been updated.

Interesting. So the next thing to do is to look into the
implementation of that. Does it allocate database resources and not
free them? Does it have internal reference loops?

Something to try: Put an explicit gc.collect() call into the loop. If
that solves your problem, you have a refloop somewhere (and you can
properly fix it by explicitly breaking the loop). If that keeps
returning large numbers, and especially if it populates gc.garbage
with a whole lot of stuff, then you definitely have refloops.

http://docs.python.org/2/library/gc.html

ChrisA

Re: script uses up all memory topic

On Fri, Mar 7, 2014 at 9:07 AM, Larry Martell <(E-Mail Removed)> wrote:
> Apparently the object returned by that call is immutable as if I look
> at id(self.tools) each pass through the loop, it is different. Is
> there some way I can recover that memory?

Not sure what mutability has to do with that. The changing id() simply
means you're getting back a new object every time. Normally, as soon
as your rebind self.tools to the new object, the old object will be
disposed of - unless it has refloops, which is what I mentioned in the
previous post, or has some other external reference.

ChrisA

Re: script uses up all memory topic

On Thu, Mar 6, 2014 at 5:11 PM, Chris Angelico <(E-Mail Removed)> wrote:
> On Fri, Mar 7, 2014 at 8:56 AM, Larry Martell <(E-Mail Removed)> wrote:
>> I figured out what is causing this. Each pass through the loop it does:
>>
>> self.tools = Tool.objects.filter(ip__isnull=False)
>>
>> And that is what is causing the memory consumption. If I move that
>> outside the loop and just do that once the memory issue goes away. Now
>> I need to figure out why this is happening and how to prevent it as
>> they do want to query the db each pass through the loop in case it has
>> been updated.
>
> Interesting. So the next thing to do is to look into the
> implementation of that. Does it allocate database resources and not
> free them? Does it have internal reference loops?
>
> Something to try: Put an explicit gc.collect() call into the loop. If
> that solves your problem, you have a refloop somewhere (and you can
> properly fix it by explicitly breaking the loop). If that keeps
> returning large numbers, and especially if it populates gc.garbage
> with a whole lot of stuff, then you definitely have refloops.
>
> http://docs.python.org/2/library/gc.html


First I added del(self.tools) before the Django call. That did not
stop the memory consumption. Then I added a call to gc.collect() after
the del and that did solve it. gc.collect() returns 0 each time, so
I'm going to declare victory and move on. No time to dig into the
Django code. Thanks.

Re: script uses up all memory topic

Chris Angelico <(E-Mail Removed)>:

> Not all problems need to be solved perfectly :) But at very least, I
> would put a comment against your collect() call explaining what
> happens: that self.tools is involved in a refloop. Most Python code
> shouldn't have to call gc.collect(), so it's worth explaining why you
> are here.

Refloops also are nothing to be avoided. Let GC do its job and forget
about it.


Marko

Re: script uses up all memory topic

On Thu, Mar 6, 2014 at 11:20 AM, Larry Martell <(E-Mail Removed)> wrote:
> On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico <(E-Mail Removed)> wrote:
>> On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell <(E-Mail Removed)> wrote:
>>> I have a script that forks off other processes and attempts to manage
>>> them. Here is a stripped down version of the script:
>>>
>>> self.sleepTime = 300
>>
>> That's not a stand-alone script.
>
> No, that is just the part that does the work (inside the 'while
> true'). I'll try and post a standalone script tomorrow.
>
>> What environment is it running in?
>
> CentOS 6,4

That's not the whole environment, though. There's a mention of Django
- does this run inside some framework?

>> Can you reproduce the problem outside of that environment?
>
> I will try that tomorrow.

Running as a stand-alone script, still under CentOS, would be what I
mean by "outside of that environment". I'm talking about making
something that can be saved to my drive and executed, perhaps with a
stubby subprocess script (eg "import time; time.sleep(86400)").

>> Also: Can you simply use multiprocessing rather than going through all
>> the effort of subprocess.Popen?
>
> Perhaps. I didn't write this. A client gave it to me and said 'figure
> out why it uses up all the memory and hangs.' I've messed around with
> for days and cannot see anything that would consume so much memory.

Ah. Yeah, that would be a fun little job to play with. My random
thought: Do the subprocesses produce spammy log output? If so, the
monitor might be collecting it all and holding it in memory in case
you want it (not knowing that you don't). The default should be to
leave them connected to your process's stdio streams, though, so that
shouldn't be the issue.

ChrisA

Re: script uses up all memory topic

On Wed, Mar 5, 2014 at 7:33 PM, Chris Angelico <(E-Mail Removed)> wrote:
> On Thu, Mar 6, 2014 at 11:20 AM, Larry Martell <(E-Mail Removed)> wrote:
>> On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico <(E-Mail Removed)> wrote:
>>> On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell <(E-Mail Removed)> wrote:
>>>> I have a script that forks off other processes and attempts to manage
>>>> them. Here is a stripped down version of the script:
>>>>
>>>> self.sleepTime = 300
>>>
>>> That's not a stand-alone script.
>>
>> No, that is just the part that does the work (inside the 'while
>> true'). I'll try and post a standalone script tomorrow.
>>
>>> What environment is it running in?
>>
>> CentOS 6.4
>
> That's not the whole environment, though. There's a mention of Django
> - does this run inside some framework?

The system this is part of uses Django, and this script makes use of
the django ORM, but it doesn't do any web stuff itself. It just kicks
off another script once for each tool found in the database, and
ensure that there's just one script pre tool running at a time, and
that no single script runs too long. The django part can easily be
removed.

>>> Can you reproduce the problem outside of that environment?
>>
>> I will try that tomorrow.
>
> Running as a stand-alone script, still under CentOS, would be what I
> mean by "outside of that environment". I'm talking about making
> something that can be saved to my drive and executed, perhaps with a
> stubby subprocess script (eg "import time; time.sleep(86400)").

Yes, I understand what you mean.

>
>>> Also: Can you simply use multiprocessing rather than going through all
>>> the effort of subprocess.Popen?
>>
>> Perhaps. I didn't write this. A client gave it to me and said 'figure
>> out why it uses up all the memory and hangs.' I've messed around with
>> for days and cannot see anything that would consume so much memory.
>
> Ah. Yeah, that would be a fun little job to play with. My random
> thought: Do the subprocesses produce spammy log output? If so, the
> monitor might be collecting it all and holding it in memory in case
> you want it (not knowing that you don't). The default should be to
> leave them connected to your process's stdio streams, though, so that
> shouldn't be the issue.

Ohh, that's a good random thought. I'll try that tomorrow and see if
that's the issue.

Re: script uses up all memory topic

On Wed, Mar 5, 2014 at 5:39 PM, Chris Angelico <(E-Mail Removed)> wrote:
> On Thu, Mar 6, 2014 at 9:27 AM, Larry Martell <(E-Mail Removed)> wrote:
>> I have a script that forks off other processes and attempts to manage
>> them. Here is a stripped down version of the script:
>>
>> self.sleepTime = 300
>
> That's not a stand-alone script.

No, that is just the part that does the work (inside the 'while
true'). I'll try and post a standalone script tomorrow.

> What environment is it running in?

CentOS 6,4

> Can you reproduce the problem outside of that environment?

I will try that tomorrow.

> Also: Can you simply use multiprocessing rather than going through all
> the effort of subprocess.Popen?

Perhaps. I didn't write this. A client gave it to me and said 'figure
out why it uses up all the memory and hangs.' I've messed around with
for days and cannot see anything that would consume so much memory.

who here uses galaxy gear. topic

who here uses a galaxy gear on thier retail note 3? i know its wrong forum but. wanted see verizon users point of view.

did you root your gear?

did make the use of your gear better with your note3?

good and bad feedbacks are welcome. thank you

CNET app uses 15% of my battery ? topic

It says CPU total 1min 50 seconds but why is it saying the GPS is on for it for over 3 hours ? I force close it and it still says its eating up my battery. Android OS and everything else minus screen is below CNET in percentage. Other than turn GPS off what can I do ? I need GPS on for other apps

Sent from my Nexus 5 using xda app-developers app