CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


Tim Hall has done it again! He has just released the 2nd edition of "Max Power".
Rather than get into details here, I urge you to check out this announcement post.
It's a massive upgrade, and well worth checking out. -E

 

Results 1 to 20 of 39

Thread: Java Process Consuming High CPU in R80

Hybrid View

  1. #1
    Join Date
    2016-10-19
    Posts
    43
    Rep Power
    0

    Default Java Process Consuming High CPU in R80

    Hi

    We are running Checkpoint Management server (R80) on a HP Proliant DL380-Gen8 server. Upon running top command, we see that Java process is consuming lot of CPU (around 90-140%) most of the time. Initially we thought it was due to log indexing which we enabled around October. But seems like that is not the root cause of it as still the situation is the same. Here are my questions:

    1) We are running Management and Logging on the same HP server, is that an issue?
    2) I saw that there was a HF from CP, is it exactly for this issue?
    3) If we decided to upgrade the CPU (currently 4 core) to 8 core, is there any documentation/guidelines available?
    4) To check the RAM on the server,I've run "cat /proc/meminfo " command. The value of memtotal is 57 Gig (57453128KB) Is that right or am i reading the output wrong?

    Please let me know is there anything that I am missing. Any help is appreciated.

    Thanks!!!

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: Java Process Consuming High CPU in R80

    Quote Originally Posted by venkata View Post
    Hi

    We are running Checkpoint Management server (R80) on a HP Proliant DL380-Gen8 server. Upon running top command, we see that Java process is consuming lot of CPU (around 90-140%) most of the time. Initially we thought it was due to log indexing which we enabled around October. But seems like that is not the root cause of it as still the situation is the same. Here are my questions:
    Java is famous for gladly consuming as much CPU and memory as you can throw at it in an attempt to run faster. :-) There are six java-based processes on a R80 SMS, you need to get the PID of the one eating the CPU from "top" or "ps -ef" then run "cpwd_admin list" to get the actual process name. Once we have that I can offer more specific advice. Log Indexing is NOT one of the java-based processes and runs with a very low process priority. Most common constant CPU-eater is SOLR which is SmartEvent event correlation and definitely will be busy if you have a lot of logs coming in from your gateways.


    1) We are running Management and Logging on the same HP server, is that an issue?
    Shouldn't be as long as you have at least 4 cores and 8GB of RAM.

    2) I saw that there was a HF from CP, is it exactly for this issue?
    I assume you are talking about Take 76, it did have some SmartLog initialization performance improvements. R80 jumbo Take 76 is highly recommended regardless, just remember to load the new version of SmartConsole too after upgrading the SMS.

    3) If we decided to upgrade the CPU (currently 4 core) to 8 core, is there any documentation/guidelines available?
    Check out the "migrate" tool if you are transitioning to new hardware. If you are dropping more cores into the same hardware nothing is required unlike a Security Gateway where you would have to redo CoreXL allocations.


    4) To check the RAM on the server,I've run "cat /proc/meminfo " command. The value of memtotal is 57 Gig (57453128KB) Is that right or am i reading the output wrong?

    Please let me know is there anything that I am missing. Any help is appreciated.

    Thanks!!!
    Post output of "free -m" please.
    Last edited by ShadowPeak.com; 2017-02-27 at 18:32.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    1,030
    Rep Power
    15

    Default Re: Java Process Consuming High CPU in R80

    Most common constant CPU-eater is SOLR which is SmartEvent event correlation and definitely will be busy if you have a lot of logs coming in from your gateways.
    SOLR is also used to index MGMT database now. In fact, R80 management uses two databases: posrtgress for holding MGMT objects and policies and SOLR to index them. Yes, I know, this is crazy as a fox.

    For the matter, I suspect a software issue, but it is still early to say. Need to see what are the processes eating CPU, as Tim mentioned.
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  4. #4
    Join Date
    2016-10-19
    Posts
    43
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Hi

    Thank you ShadowPeak & Varera for all the valuable inputs.

    You were right SOLR is the process that is consuming all the CPU (PID=6432 from output of cpwd_admin list command and ps auxwf command).
    Yes I was taking about Take 76, we are in a process of installing it and I will remember about the smart console too. . I will update if I see any better performance.

    Output of free -m command:
    total used free shared buffers cached
    Mem: 56106 55677 429 0 367 19321
    -/+ buffers/cache: 35988 20118
    Swap: 8189 857 7332


    My questions

    1) You said SOLR will consume more CPU if more logs are coming to the server, when can I say there are more logs, is there any specific optimal value based on the specs of the server?
    (Eg: Lets say If i have a 4 core processor with 32Gb RAM and I am getting around 3000 logs/sec, how do i say if that log rate is low/medium/high)

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: Java Process Consuming High CPU in R80

    Quote Originally Posted by venkata View Post
    Hi

    Thank you ShadowPeak & Varera for all the valuable inputs.

    You were right SOLR is the process that is consuming all the CPU (PID=6432 from output of cpwd_admin list command and ps auxwf command).
    Yes I was taking about Take 76, we are in a process of installing it and I will remember about the smart console too. . I will update if I see any better performance.

    Output of free -m command:
    total used free shared buffers cached
    Mem: 56106 55677 429 0 367 19321
    -/+ buffers/cache: 35988 20118
    Swap: 8189 857 7332
    Looks like 56GB of total RAM with 36GB currently being used for code execution, the rest for caching and buffering. Looks like you are just fine as far as memory, although you did manage to dip into swap space to the tune of 857MB which is a bit unexpected. Probably not anything to worry about.

    My questions

    1) You said SOLR will consume more CPU if more logs are coming to the server, when can I say there are more logs, is there any specific optimal value based on the specs of the server?
    (Eg: Lets say If i have a 4 core processor with 32Gb RAM and I am getting around 3000 logs/sec, how do i say if that log rate is low/medium/high)
    The R80 release notes have the exact specifications you are looking for on page 13, although they are for Check Point Smart-1 appliances. The closest Smart-1 to your specs is the 3050 with 2x Intel Xeon E5-2609v2 2.50GHz (QuadCore) and 32GB of RAM. If the Smart-1 3050 is set up as a R80 dedicated logging server, Check Point claims the 3050 can process 7,000 indexed logs a second. The number for a 3050 acting as both a management server and log server like yours will of course be lower. Obviously there are lots of other factors aside from CPU/memory such as I/O controller and disk speed that can impact that number.

    The indexed logs/second number for a Smart-1 3050 is higher than 7,000 for R80.10, if you sign up for the R80.10 EA you can see what it is in the R80.10 EA release notes. Not comfortable posting those numbers just yet due to EA status.
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  6. #6
    Join Date
    2011-03-29
    Posts
    28
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Hi Venkata,

    I am the technical product manager of the Check Point management products, from Check Point R&D.

    Your input about the CPU consumption is important for us and I'd like to understand it better.
    Did you experience performance issues that lead you to notice the Java CPU usage or was it just as part of a general analysis of the system?
    Have you opened a ticket or contacted Check Point about it? If so I'll appreciate to get the ticket number or relevant thread.

    I do recommend you to install the latest HF, it does include important performance improvements.

    Regarding the specific hardware requirements for your environment, I'll be happy to take it to an offline thread and assist. We can update this thread later in order to share the outcome with the community.

    If you wish to discuss the technical details, please send me an email at yaelleh@checkpoint.com.

    Thanks a lot

    Yaelle

  7. #7
    Join Date
    2006-03-08
    Location
    Lausanne
    Posts
    1,030
    Rep Power
    15

    Default Re: Java Process Consuming High CPU in R80

    Thanks, Yaelle, we would like to see more of Check Point here.
    -------------

    Valeri Loukine
    CCMA, CCSM, CCSI
    http://checkpoint-master-architect.blogspot.com/

  8. #8
    Join Date
    2016-10-19
    Posts
    43
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Hi Yaelleh,

    I did see the issue as part of regular performance monitoring & also we've faced performance issue too. Our Dashboard shows the CPU of SMS to be more than 80% most of the time. I was wondered as we got a brand new server with more capacity and we are seeing high CPU still. Its then I figured out this may be a bug with R80 and hopefully they have a HF for it.

    We are not having direct support with CP, our support is through Cadre and we have a meeting with both Cadre and CP this week about the hardware that we are having ( Whether that meets the standards that CP recommends for our environment) and installing the JHF.

    If possible, I will ask Cadre or CP to loop you in and I will surely update the thread once I have installed JHF.

    Thanks.

  9. #9
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,251
    Rep Power
    14

    Default Re: Java Process Consuming High CPU in R80

    Quote Originally Posted by venkata View Post
    Hi Yaelleh,

    I did see the issue as part of regular performance monitoring & also we've faced performance issue too. Our Dashboard shows the CPU of SMS to be more than 80% most of the time. I was wondered as we got a brand new server with more capacity and we are seeing high CPU still. Its then I figured out this may be a bug with R80 and hopefully they have a HF for it.

    We are not having direct support with CP, our support is through Cadre and we have a meeting with both Cadre and CP this week about the hardware that we are having ( Whether that meets the standards that CP recommends for our environment) and installing the JHF.

    If possible, I will ask Cadre or CP to loop you in and I will surely update the thread once I have installed JHF.

    Thanks.
    These commands should help you assess what your log/index rates are on your R80 SMS for the meeting:

    cpstat -f indexer mg
    cpstat -f log_server mg
    --
    Second Edition of my "Max Power" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  10. #10
    Join Date
    2016-10-19
    Posts
    43
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Quote Originally Posted by ShadowPeak.com View Post
    These commands should help you assess what your log/index rates are on your R80 SMS for the meeting:

    cpstat -f indexer mg
    cpstat -f log_server mg

    Hi ShadowPeak, I've used the same commands previously.

    Here are the outputs of the commands:

    Total Read Logs: 10184191882
    Total Updates and Logs Indexed: 10184191874
    Total Read Logs Errors: 0
    Total Updates and Logs Indexed Errors: 17827
    Updates and Logs Indexed Rate: 0
    Read Logs Rate: 0
    Updates and Logs Indexed Rate (10min): 0
    Read Logs Rate (10min): 0
    Updates and Logs Indexed Rate (60min): 0
    Read Logs Rate (60min): 0
    Updates and Logs Indexed Rate Peak: 7908
    Read Logs Rate Peak: 8004
    Read Logs Delay: 0
    ----------------------------------------------------------------------------

    Log Receive Rate: 9266
    Log Receive Rate Peak: 24748
    Log Receive Rate Last 10 Minutes: 9386
    Log Receive Rate Last Hour: 9536
    ---------------------------------------------------------------------------

    My questions are:

    1) Is the log receive rate (9266) in the output of cpstat -f log_server mg command, per minute or per second? The reason why I am confused is because of the values of log receive rate for 10 minutes and last hour.
    2) Are there any things that I need to worry about in the output of cpstat -f indexer mg like indexed errors value or total read logs being too high?

    Thanks in Advance!!

  11. #11
    Join Date
    2011-03-29
    Posts
    28
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Thank you very much for the update!

    I will contact Cadre to see if R&D can assist if this case.


    Thank you,

    Yaelle

  12. #12
    Join Date
    2011-03-29
    Posts
    28
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Quote Originally Posted by venkata View Post
    Hi Yaelleh,

    I did see the issue as part of regular performance monitoring & also we've faced performance issue too. Our Dashboard shows the CPU of SMS to be more than 80% most of the time. I was wondered as we got a brand new server with more capacity and we are seeing high CPU still. Its then I figured out this may be a bug with R80 and hopefully they have a HF for it.

    We are not having direct support with CP, our support is through Cadre and we have a meeting with both Cadre and CP this week about the hardware that we are having ( Whether that meets the standards that CP recommends for our environment) and installing the JHF.

    If possible, I will ask Cadre or CP to loop you in and I will surely update the thread once I have installed JHF.

    Thanks.
    Hi Again,

    Can you please let me know who in Cadre you are meeting this week?
    Also, if you have a ticket number that can be shared that will great.

    Thanks

    Yaelle

  13. #13
    Join Date
    2016-10-19
    Posts
    43
    Rep Power
    0

    Default Re: Java Process Consuming High CPU in R80

    Hello All

    I want to add update to this thread, upon a suggestion from Cadre to turn off logging for some DNS rules, that brought a significant improvement to the CPU utilization. Really helpful!!!
    Before turning off DNS logging, our CPU was at 80-95% most of the time, now it is around 55-70% most of the time.

    Also, the new log receive rate is 4795 (Previously it was around 9000, you can see the thread)
    Log Indexer rate is also very much better.

    This might be just a workaround but this is working, turning off logging on unnecessary logs (Like DNS). I have took helpful of Algosec reports too..

    Please let me know if there is something more that I can try to make it more better or something that you guys tried.

    Thanks.

Similar Threads

  1. high cpu on the fw process of the standby firewall
    By cciesec2006 in forum Miscellaneous
    Replies: 18
    Last Post: 2017-11-15, 20:23
  2. Web Applications and Active-x (java code)
    By Kiwi_wgtn in forum Mobile Access Blade (Formerly Connectra)
    Replies: 0
    Last Post: 2012-11-18, 18:52
  3. java.net.SocketException on FW1
    By sgaiotti in forum Firewall Blade
    Replies: 0
    Last Post: 2012-08-10, 06:09
  4. High CPU - FWM and CPD process (R71.30)
    By DntBrnDPig in forum Check Point UTM-1 Appliances
    Replies: 3
    Last Post: 2011-10-18, 04:03
  5. Several UTM's running high CPD process...
    By boldin in forum Check Point UTM-1 Appliances
    Replies: 0
    Last Post: 2009-02-07, 16:21

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •