Hot!Error running large script

Author
rafaesp
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2018/02/14 02:02:10
  • Status: offline
2018/02/14 02:46:41 (permalink)
0

Error running large script

I'm trying to develop a script to create a large amount of firewall policies but I'm getting the error "Broken Pipe" when the script is larger than ~260kb.
To send the script, I'm using a library available with my programming language to open a ssh connection and send the commands but the connection is broken immediately when the command is larger than those ~260kb.
 
I'm testing against FortiWiFi-60D v5.2.3.
 
Any idea of what may be happening?
Thanks a lot.
#1

6 Replies Related Threads

    emnoc
    Expert Member
    • Total Posts : 4490
    • Scores: 259
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Error running large script 2018/02/14 07:34:49 (permalink)
    0
    Are you sure it's the script or  the size? If you cut the script down in half and execute partA and then partB does it work? Do you have compression enabled ? ( if yes try turning it off ) and  the same for the option TcpKA? ( enable it or disable it and see if the output changes )
     
    I never seen scripts that big,  but have create  in excess of  3K objects and addrgrp and never had errors using bash or piping thru ssh  and various ssh clients seems to be better or less  ( i.e OpenSSH )
     
     
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #2
    ede_pfau
    Expert Member
    • Total Posts : 5380
    • Scores: 341
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Error running large script 2018/02/14 07:56:30 (permalink)
    0
    Seriously, thousands of policies on a FG-60D??
    Have you checked that against the Maximum Values Matrix, for instance here http://help.fortinet.com/fgt/56/max-values/5-6-3/max-values.html ?
    OK it says 5.000 policies hard limit but I guess you'll run into memory issues far below that.

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #3
    rwpatterson
    Expert Member
    • Total Posts : 8092
    • Scores: 163
    • Reward points: 0
    • Joined: 2006/08/08 10:08:18
    • Location: Long Island, New York, USA
    • Status: online
    Re: Error running large script 2018/02/14 08:17:04 (permalink)
    0
    The Max values matrix referenced is for a different firmware level. May be different on 5.2.x.

    -Bob - self proclaimed posting junkie!
    See my Fortigate related scripts at: http://fortigate.camerabob.com

    -4.3.18-b0689
    FGT60B
    FWF60B
    FWF80CM (2)
    FWF81CM
     
    #4
    ede_pfau
    Expert Member
    • Total Posts : 5380
    • Scores: 341
    • Reward points: 0
    • Joined: 2004/03/09 01:20:18
    • Location: Heidelberg, Germany
    • Status: offline
    Re: Error running large script 2018/02/14 08:25:18 (permalink)
    0
    might have been but isn't...

    Ede

    " Kernel panic: Aiee, killing interrupt handler!"
    #5
    emnoc
    Expert Member
    • Total Posts : 4490
    • Scores: 259
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: Error running large script 2018/02/14 09:50:09 (permalink)
    0
    So I would split the  script in  half run 2 batch per-se, that  might expose the issue(s). If the script in batch1 runs 100% and then batch2, fails it could be limits issues.
     
    As far as max-value, system log should show a message if max values are exceed. So he could run thru that and see what if any messages.
     
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #6
    rafaesp
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/02/14 02:02:10
    • Status: offline
    Re: Error running large script 2018/02/19 04:38:40 (permalink)
    0
    Thanks for your responses.
    Cutting the script in half works ok.
     
    I'm not hitting any max because I'm testing with a script that only edits one rule and sets a comment, but thousands of times. I think it demonstrates that adding more lines to the script until it reach certain size causes up the issue.

    I'm now using this library: https://github.com/spotify/pyfg/tree/master/pyFG

    And my code looks like this:
    #!/usr/bin/env python

    from pyFG import FortiOS


    if __name__ == '__main__':
        f = open('large_script.txt', 'r')
        candidate = f.read()
        f.close()

        d = FortiOS("YOUR_HOST", vdom="YOUR_VDOM", username="YOUR_USER", password="YOUT_PASSWD")
        d.open()

        print d.commit(candidate)

        d.close()

     
     
    The large file I'm using hits the max size of the forum but I upload one where you can duplicate the lines that sets the comments thousand of times until the size of the file is more than 260kb.
     
    Thanks again.
    #7
    Jump to:
    © 2018 APG vNext Commercial Version 5.5