CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


First, I hope you're all well and staying safe.
Second, I want to give a "heads up" that you should see more activity here shortly, and maybe a few cosmetic changes.
I'll post more details to the "Announcements" forum soon, so be on the lookout. -E

 

Page 1 of 2 12 LastLast
Results 1 to 20 of 26

Thread: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

  1. #1
    Join Date
    2006-07-18
    Posts
    16
    Rep Power
    0

    Default All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    All of our Edge firewalls started to lose connectivity and rebooted on their own (30+) on October 30th around 8:58 p.m. CDT. There are reports this is a worldwide issue. Anyone else having this problem?a Can't find a thing about it on Check Point's site. Big wigs want answers for downtime. Anyone know what the cause is or if it is going to happen again? Here are a couple of recent blog posts about it.

    Hurricane Labs Engineering Notes: Edge Box Reboots
    CheckPoint/Sofaware FlashForward Jack of All I.T.

    Sorry posted in the wrong forum first time. Been up to long trying to figure out what is going on.

  2. #2
    Join Date
    2007-07-12
    Posts
    143
    Rep Power
    17

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Just got this from SofaWare:
    Dear marklar,
    Hello
    I would like to update you about an issue that has been raised by the field.
    Last night we have started to get reports that edge units and safe@office units were unreachable no matter if they were managed or not.

    The issue was caused by the edge/Safe@ going into 100% cpu and rebooted a few times.
    The issue seems behind us right now as it was caused only while the clock on the unit changed from OCT 30 to OCT 31.
    We are currently working on understanding what has caused the edge/Safe@ to act that way.
    I will send another update later today
    I ended up doing a factory reset and waited a few hours, seems OK for now...

    m.
    Last edited by marklar; 2010-10-31 at 05:22. Reason: Extra info

  3. #3
    Join Date
    2006-03-24
    Location
    York, UK
    Posts
    60
    Rep Power
    19

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Hi

    We run around 300 Edge devices of which roughly half are the new N series. A quick check this morning shows they all rebooted at 2am UK time which coincides exactly with the clocks going backwards in the UK.

    We have two boxes that have not come up after the reboot (One X and one NW) but they may work once staff are available onsite to power cycle. Most of our offices are closed on Sundays.

    Trevor

    Update: Our logs clearly show that these boxes all started to power cycle every few minutes at 1am, the clocks went back at 2am and they carried on power cycling again for another hour. Once we had passed 2am for the second time they all stopped doing this. All our boxes are on 8.x firmware.
    Last edited by Trevor Rowley; 2010-10-31 at 06:25.

  4. #4
    Join Date
    2008-04-22
    Location
    Israel
    Posts
    43
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    I work for ISP/MSS and also heard from our clients that all Edges did reboot at night.All boxes returned to normal after restart. Funny :) Thanks Checkpoint for providing me with catchy header for my blog :) The D-day for CheckPoint UTM-1 Edge Appliances happened today – reboots are reported all over the world
    Taking challenges one by one.
    http://yurisk.info

  5. #5
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    19

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    The United States will "fall back" an hour next weekend. Hopefully this isn't a precursor to what will happen to our Edge units as well...

  6. #6
    Join Date
    2005-08-14
    Location
    Gig Harbor, WA, USA
    Posts
    2,500
    Rep Power
    21

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Over the weekend, Check Point received a number of reports about this issue with EDGE appliances. While the issue is still being analyzed, the initial findings suggest it is time and date specific and is not expected to happen again. More details will be provided once they are known.

  7. #7
    Join Date
    2006-04-27
    Location
    Twillight zone
    Posts
    1,010
    Rep Power
    20

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    *sigh* can confirm this as well :(
    of arround 180 units, 60 something didn't come up on their own.

    Why isn't there any official warning/information on Checkpoint support site at the time of writing?
    Only thing received atm is e-mail from Sofaware.
    Or maybe I missed it?

  8. #8
    Join Date
    2010-05-05
    Posts
    8
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    the edges devices that rebooted during the weekend is result of time and date specific and will not happen again. make sure that the clock on your Edge device is set properly, since if its configured before you will face this issue when its clock show Oct 30.

  9. #9
    Join Date
    2006-04-27
    Location
    Twillight zone
    Posts
    1,010
    Rep Power
    20

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    So almost 2 days and still no official info/entry/acknowledgment on usercenter/support pages about this.
    Guess Edge as product is either not that intresting from CP pov or bug is that embarassing to post about ;-)

  10. #10
    Join Date
    2006-02-09
    Location
    Charleston, SC
    Posts
    1,172
    Rep Power
    20

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Quote Originally Posted by abusharif View Post
    bug is that embarassing to post about ;-)
    i guess this....
    There's no place like 127.0.0.1

  11. #11
    Join Date
    2006-07-18
    Posts
    16
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    I did get a call from Check Point this afternoon confirming that it was a time/date error in their code, but not much else. I was told that they should be releasing a KB article on the episode later today. I'm surprised that the major security news outlets have not reported on this as it had to have affected many companies/users worldwide.

    We did do some testing in our lab with spare Edges and were able to duplicate the issue by setting the clock back to 10/30/2010 and 7:00 p.m. CDT. They started flaking out shortly after we reset the time with a final reboot just before 9:00 p.m. CDT. We have also set one ahead to the Saturday before next Sunday's daylight savings time change in the US. So far that one has not crashed, but it hasn't gotten to the time change just yet. I'll let you know if we encounter any issues with our testing.

  12. #12
    Join Date
    2006-07-18
    Posts
    16
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    I just looked again and the following KB article on the episode has been posted.

    https://supportcenter.checkpoint.com...a7ba5fd7-10001

    Basically, yeah they rebooted, it was a bug, it won't happen again (before they are expected to be end of support in 2015-my addition).

  13. #13
    Join Date
    2005-08-14
    Location
    Gig Harbor, WA, USA
    Posts
    2,500
    Rep Power
    21

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    The reason nothing was posted on User Center before now was we wanted to ensure we had the facts correct.

    For everyone in the Americas concerned this problem will affect them next weekend as well, the issue has nothing to do with Daylight Saving Time. The fact this bug occurred around the Daylight Saving Time switchover for Europe was coincidental.

  14. #14
    Join Date
    2010-05-05
    Posts
    8
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    there is sk56641 for explaining the issue better.

  15. #15
    Join Date
    2008-11-23
    Location
    Atlanta, GA
    Posts
    542
    Rep Power
    16

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    - boldin
    CISSP
    CCSE/R65

  16. #16
    Join Date
    2007-06-05
    Location
    Earth
    Posts
    511
    Rep Power
    18

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    I knew I wasn't crazy when our monitoring system went Defcon 5 on our dozens of monitored Edges on Saturday night...

  17. #17
    Join Date
    2006-07-18
    Posts
    16
    Rep Power
    0

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    We were able to easily duplicate the problem in the lab by setting a couple of Edges back to Oct. 30th. We also tested setting the time ahead to this weekend's daylight savings time adjustment in the US and nothing happened as far as we could tell.

  18. #18
    Join Date
    2005-08-11
    Location
    San Francisco, CA
    Posts
    1,395
    Rep Power
    20

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    It looks like this thread is becoming a bit of an Internet star. It's already picked up 25 linkbacks and we smashed our old traffic record. We've had 6,333 unique visitors in the past 24 hours.

    I'm suprised we haven't seen more about this in the mainstream media, especially given that every single Check Point Edge appliance in the world shut down simultaneously. Isn't this a pretty catastrophic failure? Maybe between the U.S. baseball World Series (the San Francisco Giants won last night, thank you very much) and today's U.S. election this story is getting crowded out. I haven't had any interview requests yet...

    Maybe some very large, very powerful customers will get pissed off enough so that Check Point will get motivated to spend some resources on QA.
    Barry J. Stiefel ("Stee-ful" or "Shtee-ful")
    B.S., MBA, CCSA/CCSE/CCSE+/CCSI
    Resilience RCSE/RCSI, Fortinet FCSE
    CISSP, MCSE, NSA ISM
    Founder of CPUG
    Founder of CPUG University

  19. #19
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    19

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Check Point's competitors certainly have picked up plenty of anti-Check Point fodder for their sales presentations due to this incident and the Zeus Trojan scareware popup in ZoneAlarm about a month and a half ago. If something like this latest incident occurred on their high-end firewalls that run SecurePlatform/Linux or Nokia IPSO it would be truly disastrous.

  20. #20
    Join Date
    2006-12-04
    Posts
    1,316
    Rep Power
    19

    Default Re: All Edge firewalls rebooted 10/30/2010 8:58 p.m.

    Some times it is better to invest some ammount of money for QA and TAC than lost them (money, customers ...) .....
    Last edited by serlud; 2010-11-03 at 09:11.

Page 1 of 2 12 LastLast

Similar Threads

  1. New! R70 IPv6Pack (2010-08-12)
    By melipla in forum IPv6
    Replies: 2
    Last Post: 2010-08-25, 12:27
  2. Vpn solution requirement over ADSL with UTM EDGE/safe office firewalls
    By sebastan_bach in forum Check Point UTM-1 Edge Appliances
    Replies: 5
    Last Post: 2010-03-27, 08:40
  3. Cluster FW rebooted unintentionally?
    By JackYi in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 3
    Last Post: 2006-08-25, 04:18
  4. Cluster FW rebooted unintentionally
    By JackYi in forum Check Point IP Appliances and IPSO (Formerly Sold By Nokia)
    Replies: 0
    Last Post: 2006-08-22, 13:51

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •