Hot!Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours

Author
GabbaTech
New Member
  • Total Posts : 3
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/02/18 10:52:00
  • Status: offline
2019/02/18 11:11:20 (permalink)
0

Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours

Hi,
 
I'm hoping someone can assist me. I've got an analyzer (3000D) that's been upgraded from version 5.6 to 6.0 (currently running 6.0.3)
 
The box will complete it's database rebuild as per the documentation but after about 8 hours of running, most of the processes go into a zombie state and the web gui becomes unavailable. You can ssh onto the box but an execute reboot just sits there and you have to physically power cycle the device. I've tried completely removing the db and rebuilding it but I continue to have the same issues.
 
Are there any commands that can be used to see what the box is actually doing (something like view /var/log/messages) or something similar. The diagnose test application commands sometimes work and sometimes don't and not having access to the actual unix cli is quite frustrating.
 
The box is swapping a lot and execute iotop shows sqllogd being the main culprit. It looks like the box is running out of memory
 
Total DISK READ :    1062.73 K/s | Total DISK WRITE :     221.40 K/s
Actual DISK READ:    1059.04 K/s | Actual DISK WRITE:     571.95 K/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO<    COMMAND
 7240 be/4 root      856.09 K/s    0.00 B/s 99.74 %  0.00 % sqllogd
 
execute top again shows sqllogd using lots of memory and the box swapping
 
top - 21:08:35 up  1:06,  0 users,  load average: 10.17, 11.30, 11.31
Tasks: 369 total,   6 running, 360 sleeping,   0 stopped,   3 zombie
%Cpu(s):  2.9 us, 11.9 sy,  0.0 ni, 78.8 id,  6.4 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 16013.96+total,   92.070 free, 15234.69+used,  687.203 buff/cache
MiB Swap: 16012.99+total, 10370.72+free, 5642.270 used.   33.504 avail Mem

  PID USER      PR  NI    VIRT    RES  %CPU %MEM     TIME+ S COMMAND
 7240 root      20   0 16.038g 0.014t   0.7 89.8   3:57.35 D /bin/sqllogd
 
If anyone has any advice (other than completely wiping the box and starting again), I'd gladly appreciate it.
 
Cheers,
 G
#1

5 Replies Related Threads

    RCHEN_FTNT
    New Member
    • Total Posts : 3
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/08/28 10:38:34
    • Status: offline
    Re: Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours 2019/02/19 10:55:27 (permalink)
    0
    Hi Gabba,
     
    It seems it hit the bug#502046(initd has a bug to deadlock in certain situations and stop responding to
    signals/respawing crashed daemons) which was fixed on B279. and it had sqllogd high memory issue, Is it possible to get some more info as below for investigation?
     
    diag deb klog
    diag log device
    diag test app  oft 2
    diag fortil lograte
    diag test app sqll 2
    diag test app sqll 205 stats
    diag test app sqll 206 stats
    diag sql st sqlp
    diag deb cr  rea
     
    Thanks!
     
    Rosie
     
     
    #2
    GabbaTech
    New Member
    • Total Posts : 3
    • Scores: 0
    • Reward points: 0
    • Joined: 2019/02/18 10:52:00
    • Status: offline
    Re: Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours 2019/02/19 11:55:22 (permalink)
    0
    Hi Rosie,
     
    Thanks very much for the reply.
     
    I've been watching the box over the last 24 hours and it's definitely just before 8 hours during the database rebuild that the box begins to swap and inserts into the database start taking up to 30 minutes. The only way to resolve is to disable sql, reboot and re-enable sql.I've now removed the db and restarted and just let the box run without rebuilding from the archive logs.
     
    Is there an email address or something I can send the output of those commands to as diag log device for example will include firewall hostnames and serial numbers which I don't really want floating around on the internet :)
     
    Regards,
     G
    #3
    RCHEN_FTNT
    New Member
    • Total Posts : 3
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/08/28 10:38:34
    • Status: offline
    Re: Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours 2019/02/19 14:56:26 (permalink)
    0
    Hi Gabba,
     
    It is better to  open a ticket for these mentioned  debug info.
     
    Regards,
     
    Rosie
    #4
    RCHEN_FTNT
    New Member
    • Total Posts : 3
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/08/28 10:38:34
    • Status: offline
    Re: Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours 2019/02/19 15:03:25 (permalink)
    0
    Hi Gabba,
     
     
    It is better to open a ticket to upload the debug info.
     
    Regards,
     
    Rosie
    #5
    GabbaTech
    New Member
    • Total Posts : 3
    • Scores: 0
    • Reward points: 0
    • Joined: 2019/02/18 10:52:00
    • Status: offline
    Re: Analyzer upgrade from 5.6 to 6.0 becomes unresponsive after 8 hours 2019/02/19 20:52:04 (permalink)
    0
    Thanks Rosie. I don't think the device is under maintenance though with it being an old 3000D model.
     
    Appreciate your assistance.
     
    Regards,
     G
    #6
    Jump to:
    © 2019 APG vNext Commercial Version 5.5