Hot!SSL Inspection policy failing to use new certificate

Author
CodeMonkey
Silver Member
  • Total Posts : 114
  • Scores: 14
  • Reward points: 0
  • Joined: 2015/01/12 03:18:57
  • Status: offline
2017/11/16 07:55:11 (permalink)
0

SSL Inspection policy failing to use new certificate

Fortigate 60D v5.2.11
 
We've had cause to re-issue the certificate that we use for deep inspection on outbound traffic (moving from SHA-1 to SHA-256). This certificate has been installed as trusted on all affected internal clients.
 
To try and ensure a smooth transition, I've installed the new certificate and set it for use in a cloned copy of the outbound deep inspection SSL inspection policy, leaving the original SHA-1 certificate used by the original outbound inspection policy.
 
I've switched one of the firewall rules for traffic to use the new SSL inspection policy with the SHA-256 certificate. 
That was 50 minutes ago, however the fortigate is still utilising the original SHA-15 certificate for inspection (as checking any inspected HTTPS website certificate chain confirms). This behaviour is replicated across multiple clients and browsers (IE11, Chrome, Firefox).
 
I'm assuming that the Fortigate has cached the old certificate. 
Rebooting the firewall is not an option.
How do I 'encourage' the firewall to respect this configuration change and begin using the new SHA-256 certificate to inspect outbound traffic? Is there a process I can restart without interfering with other functionality / traffic?
 
 
#1

18 Replies Related Threads

    tanr
    Gold Member
    • Total Posts : 389
    • Scores: 12
    • Reward points: 0
    • Joined: 2016/05/09 17:09:43
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/16 08:14:00 (permalink)
    0
    I think clearing the ssl certificate cache is done with
     
    diag test app ssl 12
     
    Details at http://kb.fortinet.com/kb/viewContent.do?externalId=FD32679
     
    #2
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/16 08:52:07 (permalink)
    0
    Unfortunately 
    diagnose test application ssl 0
    is rejected as a command parse error.
     
    Substituting 'debug' for 'test' and trying 0, 12, and 99 options returns no error in the CLI and no output.
    Symptoms remain unchanged with the certificate chain showing the 'old' SHA1 certificate.
     
     
    I'm not sure the ssl application is present anymore (per https://forum.fortinet.com/tm.aspx?m=135228)
     
    I've also tried restarting proxyworker, with no change to the symptoms. Very frustrating.
     
     
    #3
    emnoc
    Expert Member
    • Total Posts : 4404
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/16 09:18:53 (permalink)
    0
    Can you  do any of the following
     
     
    1: disable the ssl-inspection , wait and then re-enable
     
    or
     
    2: disable and  then re-enable the  fwpolicy-id <xxx>
     
    Does that fix the issue?
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #4
    tanr
    Gold Member
    • Total Posts : 389
    • Scores: 12
    • Reward points: 0
    • Joined: 2016/05/09 17:09:43
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/16 09:19:19 (permalink)
    0
    Ah, sorry - didn't realize that was out of date.
    Don't suppose it's in the wanopt webcache?
    #5
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 00:10:10 (permalink)
    0
    emnoc
    Can you  do any of the following
     
     
    1: disable the ssl-inspection , wait and then re-enable
     
    or
     
    2: disable and  then re-enable the  fwpolicy-id <xxx>
     
    Does that fix the issue?
     


    Tried 1 yesterday (which I should have mentioned - apologies) - removing the ssl-inspection for about 1/2 hour made no difference, with HTTPS browsing still intercepted and re-encrypted using our certificate.
     
    I've tried 2 this morning after coming in early - no dice.
     
    I've now also tried removing SSL inspection from the policy, deleting the new (cloned) inspection object and recreating it from scratch in case there was an issue with cloning.
    After applying the fresh SSL inspection object to the policy the chain still shows the old certificate in use.
    #6
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 00:12:40 (permalink)
    0
    tanr
    Ah, sorry - didn't realize that was out of date.
    Don't suppose it's in the wanopt webcache?


    No worries.
     
    I don't know anything about wanopt webcache I'm afraid - the 60D doesn't support SSL Offloading AFAIK (which I believe is related based on a very quick search?).
    I find no reference to it in a config backup.
    post edited by CodeMonkey - 2017/11/17 00:15:12
    #7
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 01:35:31 (permalink)
    0
    At this point I've:
    - updated the original ssl inspection policy to point to the new (sha256) certificate
    - deleted the old (sha1) certificate from the FGT appliance.
    - removed the new inspection policy.
     
    Yet all indications show that the FGT appliance continues to use the original cert.
     
    *Update*
    It seems that I need to locate the 5.2.x branch equivalent of clearing the ssl certificate cache
    diagnose test application ssl 12

     
    We are seeing the new certificate on sites not (recently) visited, however the majority of sites still show the old certificate in the chain.
    Obviously I've ensure that browser caches are cleared etc. - this behaviour can be seen on multiple clients, in multiple browsers and persists through reboots etc. hence my seeking a FGT cache clearing mechanism.
    post edited by CodeMonkey - 2017/11/17 04:00:32
    #8
    packetpusher
    Silver Member
    • Total Posts : 85
    • Scores: 0
    • Reward points: 0
    • Joined: 2016/07/18 08:44:14
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 10:44:05 (permalink)
    0
    Did you solve the issue?
    #9
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 10:48:35 (permalink)
    0
    mstoyanoff
    Did you solve the issue?

    Nope.

    Still needing to find out how to clear the cache - symptoms still as last described.
    I'm UK based and have left it over the weekend in case it's time based.
    All suggestions still welcome.
    #10
    emnoc
    Expert Member
    • Total Posts : 4404
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 11:35:11 (permalink)
    0
    Dumb question, is the ssl-certificate removed from the  appliance?
     
     
    What does diag debug flow show ?
     
     
    I just test my own firewall and nothing is being cached btw.
     
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #11
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 11:56:58 (permalink)
    0
    emnoc
    Dumb question, is the ssl-certificate removed from the  appliance?
     
     
    What does diag debug flow show ?
     
     
    I just test my own firewall and nothing is being cached btw.



    Yes - 100% removed as of about 10am this morning.
    No diag debug flow output as I'm not at work until Monday morning .
     
    When browsing to inspected sites, their certificate chain still shows the old (removed) certificate some 6 hours+ after it was removed from the appliance. Removing the old certificate from trusted root store on the client results in non-functional browsing.
    I've cleared every cache known to man on my laptop and the issue persists, however I found one or two (seldom visited) HTTPS sites using the new certificate. This is what lead me to think caching might be at play, along with the previous existence of a CLI command specifically to clear the SSL cache.
     
    All affected machines currently have both old and new certificates in their trusted store, so I believe nothing should break while this state continues.
    Will have to see if time cures it, otherwise I may have a small window to reboot later in the week if I can get to the office early enough. 
    #12
    emnoc
    Expert Member
    • Total Posts : 4404
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/17 17:48:05 (permalink)
    0
    Okay your probably right and it'c cachcing on the  browser can you do  the following
     
    1: find a macosx and  run gnutls-cli <site name> does the  old chain or new chain shows up?
     
    2: on a browser FIREFOX  do  refresh  ( here's link
     
    https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings?redirectlocale=en-US&redirectslug=reset-firefox-easily-fix-most-problems
     
    3: now run a site and do you see the old or new chain?
     
    4: if you have chrome they have an optional to refresh also
     
    try that but the fortigate is not the culprit in this case. Report back on what you discovery using step #1 or #2 or #3
     
    Thanks
     
    Ken
     

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #13
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/20 00:33:07 (permalink)
    0
    emnoc
    Report back on what you discovery using step #1 or #2 or #3
     



    All browser caches were cleared during testing, however..
    1. No mac os available
    2. Firefox (which I have installed but only use for testing internally developed sites and services) reset completed and certificates reimported as trusted into Firefox. Firefox continues to use the SHA1 certificate that is no longer present on the FGT appliance.
     
    However, if I pick a page never before visited, e.g. https://support.mozilla.org/en-US/questions/995117 then the certificate chain uses the new SHA256 certificate.
     
    I've also dug out an unused spare laptop and tried accessing sites from it that I have accessed before from my usual laptop (but not the spare); these also show the SHA1 certificate in the chain. Accessing sites never before visited show the new SHA256 certificate.
     
    I think at this point I'm going to have to find a way to schedule a reboot of the FGT over the course of the week - my suspicion is that this will solve the problem; I came across some old configuration notes from 5.2.2 / 5.2.3 that imply I had this issue previously and resolved it only with a reboot.
    #14
    emnoc
    Expert Member
    • Total Posts : 4404
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/20 06:57:21 (permalink)
    0
    confused
     
    So the FF reset did not work or you didn't do if?
     
    If you take a brand new "laptop / desktop" and connect  it thru the fortigate does it show the correct cert  in the chain?
     
    Your problem is not the  FGT btw, think about it. If the cert is out of the SSL-inspection  and a client is  seeing the old cert, it's cached by the client.
     
    If you  don't have have gnutls-cli than use  curl ( it's available on  windows but my exprience is limited in windoze env ;) )
     
    read this blog , it's a BC-SGproxy but the same principle applies. We have various  clients using  the proxy and various sites and client had the trust-EnterpriseOrgCA in the chain. I even use forum.fortinet.com in my example ;)
     
    http://socpuppet.blogspot.com/2017/11/ssl-cert-caching-for-mitm-inspections.html
     
     BTW: I should re-label the title  since the caching is not the MiTM device , it's the client.
     
    post edited by emnoc - 2017/11/20 06:58:25

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #15
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/20 07:16:00 (permalink)
    0
    emnoc
    So the FF reset did not work or you didn't do if?
     

    As above, I did the reset in Firefox. It made no difference to the behaviour.
     
    emnoc
    If you take a brand new "laptop / desktop" and connect  it thru the fortigate does it show the correct cert  in the chain?

    I have no more machines to test with.


    emnoc
    Your problem is not the  FGT btw, think about it. If the cert is out of the SSL-inspection  and a client is  seeing the old cert, it's cached by the client.

    I'm sure you're right, however I have cleared all accessible browser caches in my attempts to resolve this and a Firefox installation reset per instructions still exhibits the same behaviour as before. 
    I am now out of time to spend on this issue for the moment, have no additional resources to test with, and no opportunity to reboot the FGT before planned maintenance early next month.
    If the problem is some hidden and inaccessible caching on user machines I would hope that it would clear automatically before then. Failing that, I'll see what a reboot brings.
    #16
    emnoc
    Expert Member
    • Total Posts : 4404
    • Scores: 249
    • Reward points: 0
    • Joined: 2008/03/20 13:30:33
    • Location: AUSTIN TX AREA
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/20 12:08:48 (permalink)
    0
    "browser caches"
     
    This is not a browser  cache issues. It's CA-store for website clearing the browser does not fix this in  FF.
     
     
    What happens when your  hit the site the web-browser checks the  CAchain and any OSCP/CRL issues and caches this. This cause the  "servercert+CAchain issuer" to be cache and no more looks. Chrome is different, I don't use  MSIE so can't speak 100% on that browser.
     
    When you remove the MiTM  device out of the picture ( your fortigate , or in my case  BCPROXYSG ) the  cache CAissues is falsely  display by the "web-client-browser". This is why tools like GNUtls and Curl are helpful.
     
     
    Reseting the  browser SSL-CA cache or forcing the  web-browser to perform a look and not cache  could be used.
     
     
    Try this, the system(s) you have problems on ( a laptop )
     
      1; While on the network,  that has  the SSL inspection,  goto a website that you never been to....
     
      2; read the browser ca-chain ( it will shows the CA certificate that you  trust and the MiTM devices uses .) Right ?
     
      3; now take that  LAPTOP device down  and goto off-network   Wifi/Mifi,starbuck,homedepot,walmart and go to the same  exact  website ? The CAchain and issuer will be the same value you have notice in step#2 .Right ? and with your  certificate from the  MITM device.
     
     
      4; You can also see this behavior  in FF, by going to security >  certificates > and reviewing the site certificate that are cache
     
      5;  in Chrome ( which exhibits the right ca-chain always  regardless if your  on or off-net  ) you go to developer-tool > security,  and click the certificate details. It will always show you the  correct chain for if your on-net with a MiTM device or not from my experiences
     
    Again it's not the MiTM it's how your browser cache  SSL-chain/issue details and what inspect if the device is on or off-net.
     
     
    Ken

    PCNSE6,PCNSE7, ACE, CCNP,FCNSP,FCESP,Linux+,CEH,ECSA,SCSA,SCNA,CISCA email/web
    #17
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/11/21 00:23:02 (permalink)
    0
    emnoc
    Try this, the system(s) you have problems on ( a laptop )
     
      1; While on the network,  that has  the SSL inspection,  goto a website that you never been to....
     
      2; read the browser ca-chain ( it will shows the CA certificate that you  trust and the MiTM devices uses .) Right ?
     
      3; now take that  LAPTOP device down  and goto off-network   Wifi/Mifi,starbuck,homedepot,walmart and go to the same  exact  website ? The CAchain and issuer will be the same value you have notice in step#2 .Right ? and with your  certificate from the  MITM device.

     
    Unfortunately, using Chrome and our guest wifi (that sits outside the FGT's sphere of influence) and testing with https://www.bbc.co.uk that is not what I see when I follow the above.
     
    BBC site is chained from our original SHA1 cert when visited inside from behind the FGT. After switching to guest wifi and reloading the BBC site I see the chain terminating against a GlobalSign cert.
     
    Regardless I am out of time to mess with this situation. Thank you for your assistance Ken.
    I will update this thread again when I get more time to look at this again. For the moment it is an irritant but no a show-stopper (as we have both old SHA1 and new SHA256 loaded on all affected machines). FGT reboot is scheduled for early next month in our next maintenance window.
    post edited by CodeMonkey - 2017/11/21 00:26:00
    #18
    CodeMonkey
    Silver Member
    • Total Posts : 114
    • Scores: 14
    • Reward points: 0
    • Joined: 2015/01/12 03:18:57
    • Status: offline
    Re: SSL Inspection policy failing to use new certificate 2017/12/04 00:07:16 (permalink)
    0
    To update with a resolution.
     
    The issue persisted until we performed a reboot on the Fortigate appliance during our maintenance window yesterday.
    This immediately resolved the issue across all machines for all external sites (including those previously visited).
    #19
    Jump to:
    © 2017 APG vNext Commercial Version 5.5