{"id":556,"date":"2021-02-26T13:55:58","date_gmt":"2021-02-26T19:55:58","guid":{"rendered":"http:\/\/computing.fnal.gov\/lqcd\/?page_id=556"},"modified":"2021-09-21T09:46:58","modified_gmt":"2021-09-21T14:46:58","slug":"troubleshooting-kerberos-kinit-problems","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/lqcd\/troubleshooting-kerberos-kinit-problems\/","title":{"rendered":"Kerberos and SSH troubleshooting"},"content":{"rendered":"\n

Troubleshooting Kerberos kinit problems<\/h3>\n\n\n\n
Error message:<\/span> kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED<\/span><\/pre>\n\n\n\n
Problem:<\/strong> <\/strong>Your Kerberos account has expired.<\/p>\n\n\n\n
Solution:<\/strong> Please see the User Accounts web page<\/a> for more information on renewing your Kerberos account.<\/p>\n\n\n\n
Error message: kinit(v5): Cannot find KDC for requested realm while getting initial credentials<\/span><\/pre>\n\n\n\n
Problem: <\/strong>\/etc\/krb5.conf<\/code> file does not contain .FNAL.GOV<\/code> information.<\/p>\n\n\n\n
Solutions:<\/strong><\/p>\n\n\n\n
  1. Replace \/etc\/krb5.conf<\/code> with your OS-specific Fermilab-supplied version of krb5.conf<\/a>.<\/li>
  2. Modify \/etc\/krb5.conf<\/code> adding Fermilab-specific stanzas as instructed in the User Accounts web page<\/a>.<\/li>
  3. If you do not have permission to modify \/etc\/krb5.conf<\/code>, copy Fermilab-supplied version<\/a> into your home area, and execute export KRB5_CONFIG=$HOME\/krb5.conf<\/code> to tell all Kerberos commands to use the user’s copy of krb5.conf<\/code>.<\/li><\/ol>\n\n\n\n
    Related problem:<\/strong><\/p>\n\n\n\n
    On Macintosh computers, Kerberos is installed on all recent versions. However, there are two locations and names for krb5.conf,<\/p>\n\n\n\n
    \/etc\/krb5.conf<\/p>\n\n\n\n
    and<\/p>\n\n\n\n
    \/Library\/Preferences\/edu.mit.Kerberos<\/p>\n\n\n\n
    (Note: the file in \/Library is named edu.mit.Kerberos, not krb5.conf.) Either will work, but you should only have one.<\/p>\n\n\n\n
    Error message: kinit: Unable to acquire credentials for 'user@FNAL.GOV': Cannot contact any KDC for realm 'FNAL.GOV'<\/span><\/pre>\n\n\n\n
    Problem:<\/strong> You are behind a firewall or are using an internet connection which has a “NAT” (Network Address Translation), such as on a home wireless router.<\/p>\n\n\n\n
    Solutions:<\/strong><\/p>\n\n\n\n
    Step 1:<\/strong> Check your connectivity, as shown below, to one of the Fermilab Kerberos authentication servers (such as krb-fnal-1.fnal.gov<\/code>) to make sure you can reach the server at the other end. If successful move to step 2. If fail, send us email at lqcd-admin@fnal.gov<\/a>.<\/p>\n\n\n\n
    [@mylaptop ~]$ telnet krb-fnal-1.fnal.gov 88\nTrying 131.225.110.105...\nConnected to krb-fnal-1.fnal.gov.\nEscape character is '^]'.\n^]\n \ntelnet> quit\nConnection closed.<\/pre>\n\n\n\n
    OR, in case of some mac OS versions that are missing the telnet<\/code> utility, use the nc<\/code> utility as follows:<\/p>\n\n\n\n
    [@mylaptop ~]$ nc -vz krb-fnal-1.fnal.gov 88\nConnection to krb-fnal-1.fnal.gov port 88 [tcp\/kerberos] succeeded<\/pre>\n\n\n\n
    Step 2: <\/strong>Request an address-less Kerberos ticket as follows:<\/p>\n\n\n\n
          kinit -a username@FNAL.GOV<\/p>\n\n\n\n
    If you do<\/p>\n\n\n\n
          klist -a<\/p>\n\n\n\n
    you should see as the last line<\/p>\n\n\n\n
         Addresses: (none)<\/p>\n\n\n\n
    Error message: kinit: Preauthentication failed while getting initial credentials<\/span><\/pre>\n\n\n\n
    Problem:<\/strong> kinit<\/code> fails with preauthentication error.<\/p>\n\n\n\n
    Solutions:<\/strong><\/p>\n\n\n\n
    1. Usually the problem is simply that you have typed in your kerberos password incorrectly. Please try again.<\/li>
    2. If you have lost your kerberos password, call the Fermilab Service Desk<\/a> at (630) 840 2345, during business hours to have the password reset.<\/li>
    3. Occasionally this is not a password problem, but a problem with your system’s clock. Make sure that the date<\/code> command returns a time correct to within 5 minutes.<\/li><\/ol>\n\n\n\n
      Error message: kinit: krb5_get_init_creds: Too large time skew<\/span><\/pre>\n\n\n\n
      Problem:<\/strong> kinit fails with time skew message.<\/p>\n\n\n\n
      Solution:<\/strong><\/p>\n\n\n\n
      Your system clock must be within 5 minutes of the correct value.<\/p>\n\n\n\n
      Error message: kinit: KDC has no support for encryption type while getting initial credentials<\/span><\/pre>\n\n\n\n
      Problem:<\/strong> kinit fails with complaint about encryption type.<\/p>\n\n\n\n
      Solutions:<\/strong><\/p>\n\n\n\n
      This problem appears on recent Ubuntu and related Linux distributions. To fix, edit \/etc\/krb5.conf<\/code> file, and in the [libdefaults]<\/code> section add<\/p>\n\n\n\n
      allow_weak_crypto = true<\/p>\n\n\n\n
      Error message: kinit: Client not found in Kerberos database while getting initial credentials<\/span><\/pre>\n\n\n\n
      Problem:<\/strong> kinit<\/code> fails with database complaint.<\/p>\n\n\n\n
      Solutions:<\/strong><\/p>\n\n\n\n
      1. Your kerberos principal may differ from your username on your local system. Use kinit username@FNAL.GOV<\/code> where username is your Fermilab kerberos principle. kinit<\/code> without options will default to using your local username.<\/li>
      2. Your Fermilab ID or Visitor ID has expired. Please see the User Accounts web page<\/a> for more information on renewing your Fermilab ID.<\/li><\/ol>\n\n\n\n
        Error message: kinit: Client's entry in database has expired<\/span><\/pre>\n\n\n\n
        Problem:<\/strong> kinit<\/code> fails because of an expired password<\/p>\n\n\n\n
        Solutions:<\/strong><\/p>\n\n\n\n
        1. You must change your kerberos password once a year with the kpasswd<\/code> command. You can try to change your password, even if it is expired, by using kpasswd<\/code> on your local machine. More detailed instructions are available here<\/a>.<\/li>
        2. You can also call the Fermilab Service Desk at (630) 840 2345, during business hours to have the password reset.<\/li><\/ol>\n\n\n\n
          Error message: kinit: Password incorrect<\/span><\/pre>\n\n\n\n
          Problem:<\/strong> If you are sure your Kerberos password is correct but you are on a MAC OS 10.10 (Yosemite) kinit<\/code> will fail because the Kerberos pass phrase is DES encoded, which Yosemite no longer accepts.<\/p>\n\n\n\n
          Solutions: <\/strong><\/p>\n\n\n\n
          You will need to reset your Kerberos password as follows:<\/p>\n\n\n\n
          1. You have access to a non-Yosemite machine with Kerberos client software installed, do kinit<\/code> to obtain a Kerberos ticket and then SSH to one of our head nodes, for example lq.fnal.gov<\/code>. Once there do a kpasswd<\/code> to change your Kerberos password. Alternatively, you may call the Fermilab Service Desk at (630) 840 2345, during business hours and request that they reset your kerberos password. <\/li>
          2. Update the krb5.conf<\/code> file on your Yosemite machine with the latest version from Fermilab-supplied version posted here<\/a>.<\/li>
          3. On your Yosemite machine, go ahead and attempt kinit<\/code> and ssh<\/code>, which should now work.<\/li><\/ol>\n\n\n\n
            Troubleshooting SSH problems<\/h3>\n\n\n\n
            Error message: ssh_dispatch_run_fatal: Connection to 131.225.202.32: unexpected internal error<\/span><\/pre>\n\n\n\n
            Problem:<\/strong> You may have just upgraded to the latest Mac OS X version or changed the SSH options on your client.<\/p>\n\n\n\n
            Solution:<\/strong>\u00a0Try to “ssh -o GSSAPIKeyExchange=no lq.fnal.gov<\/code>”\u00a0to one of our servers. If it works then add “GSSAPIKeyExchange no<\/code>”\u00a0to your SSH client config file (e.g. \/etc\/ssh\/ssh_config<\/code>).<\/p>\n\n\n\n
            See also<\/strong>: The CMS support group has posted some information specifically addressing the upgrade to Mac OS Big Sur. Visit their troubleshooting page <\/a>for more information on changes with Big Sur and issues with Anaconda affecting Kerberos authentication.<\/p>\n\n\n\n
            Error message: permission denied<\/span><\/pre>\n\n\n\n
            SSH login failures will be indicated by a permission denied message. If none of the solutions below fixes your problem please email the output of the command “ssh -vvv lq.fnal.gov<\/code>” to lqcd-admin@fnal.gov<\/a> for further assistance.<\/p>\n\n\n\n
            Problem:<\/strong> Kerberos client and SSH using different credential cache file locations.<\/p>\n\n\n\n
            Solution:<\/strong> We have mostly encountered this on MAC 10.9.x versions where Kerberos clients are installed from two different sources. In such a situation Kerberos client binaries end up in \/opt\/local\/bin and in \/usr\/bin. Use the Kerberos client kinit installed in \/usr\/bin to obtain a Kerberos ticket. Also make sure there is a subdirectory .ssh in your home directory. Make sure the subdirectory has a file named config with the following lines:<\/p>\n\n\n\n
            Host *.fnal.gov\nGSSAPIAuthentication yes\nGSSAPIDelegateCredentials yes\nGSSAPITrustDNS yes<\/pre>\n\n\n\n
            Problem:<\/strong> Not having a kerberos ticket granting ticket (TGT), or having an expired TGT.<\/p>\n\n\n\n
            Solution:<\/strong> Verify with the klist -f command that you have a ticket. If you don’t have a ticket, or have an expired ticket, get a new ticket with kinit as follows:<\/p>\n\n\n\n
            lqcdp4ee:~$ kinit -fr 7d username@FNAL.GOV\nlqcdp4ee:~$ klist -f\nTicket cache: \/tmp\/krb5cc_1234\nDefault principal: username@FNAL.GOV\n \nValid starting \u00b7\u00b7 Expires\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7 Service principal\n08\/17\/12 09:31:16 08\/18\/12 11:31:16 krbtgt\/FNAL.GOV@FNAL.GOV\nrenew until 08\/24\/12 09:31:09, Flags: FRIA<\/pre>\n\n\n\n
            Normal output, indicating that a forwardable, renewable, ticket exists. Check the expiration time – if the current time is past the expiration, login attempts will fail.<\/p>\n\n\n\n
            lqcdp4ee:~$ klist -f\nklist: No credentials cache file found (ticket cache \/tmp\/krb5cc_5598)<\/pre>\n\n\n\n
            If you see the above message you do not have a Kerberos ticket. Use kinit to get a ticket before attempting to login. Kerberos tickets expire after 24 hours. If you include the -r 7d switch on your kinit command line, you will receive a renewable ticket. Renewable tickets may be renewed by typing kinit -R before they expire at the end of any 24 hour period. Tickets are renewable for up to the period specified in the -r switch, to a maximum of 7 days.<\/p>\n\n\n\n
            Another useful switch to kinit is -f, which asks for a forwardable ticket. If you have a forwardable ticket, once you login to a Fermilab machine, say lq.fnal.gov, you can then login from lq to another machine without executing a new kinit. It is in general a bad idea to use kinit on any machine but your local system, as your password may be captured as it traverses the internet. The only time typing a kinit password is safe on a remote machine is when you are using an encrypted connection, like with ssh.<\/p>\n\n\n\n
            Problem:<\/strong> Not having an account on the target machine, or having an account on the target machine under a different username.<\/p>\n\n\n\n
            Solution:<\/strong> A permission denied error will occur if you do not have an account on the target machine, or if your username on the target machine differs from your username on your local machine. Try<\/p>\n\n\n\n
            ssh username@lq.fnal.gov<\/p>\n\n\n\n
            or<\/p>\n\n\n\n
            ssh -l username lq.fnal.gov<\/p>\n\n\n\n
            where username is your Fermilab username (the same name that you used in your kinit command). If this fails, send e-mail to lqcd-admin@fnal.gov<\/a> and ask that the administrators verify that you have a valid account on the Fermilab lattice QCD systems.<\/p>\n\n\n\n
            Problem: <\/strong>Using an internet connection which has a “NAT” (Network Address Translation), such as on a home wireless router<\/p>\n\n\n\n
            Solution:<\/strong> Nearly all home routers, wired or wireless, have a “NAT” function, which results in your local system having a different local network address than what is presented to remote machines. This allows you to have multiple local machines and only one external IP address. Your local addresses will generally be something like 192.168.X.Y, or 10.X.Y.Z, when a NAT is present.<\/p>\n\n\n\n
            With a NAT, your ssh logins may fail with Incorrect net address. To fix this, use “address-less” tickets. First, use kdestroyto delete your current ticket. Then, use kinit with -a, -A, or -n to request an address-less ticket. The switch required varies with kerberos versions, so use man kinit on your local system to determine which of these three switches to use.<\/p>\n\n\n\n
            For Mac OS users, please be aware that the default behaviour on Mac OS is to supply address-less tickets, so you should also be able to simply drop the -A or -a switch entirely.<\/p>\n\n\n\n
            Problem:<\/strong> Using an SSH client which does not have Kerberos authentication enabled<\/p>\n\n\n\n
            Solution:<\/strong> Some versions of ssh will not attempt to perform kerberos authentication. In this case, you will receive a permission denied error. To enable kerberos authentication, try the following -o switch:<\/p>\n\n\n\n
            ssh -o “GSSAPIAuthentication yes” username@lq.fnal.gov<\/p>\n\n\n\n
            The quotation marks are required. If this form of SSH succeeds, you can configure your local system to always attempt to use kerberos authentication by editing either $HOME\/.ssh\/config or \/etc\/ssh\/ssh_config and adding these lines:<\/p>\n\n\n\n
            Host *.fnal.gov\nGSSAPIAuthentication yes\nGSSAPIDelegateCredentials yes<\/pre>\n\n\n\n
            The GSSAPIDelegateCredentials line is necessary if you want to use X-windows clients on the remote (Fermilab) system. Note that you may also need a -X or -Y switch on your ssh command to enable X forwarding.<\/p>\n","protected":false},"excerpt":{"rendered":"
            Troubleshooting Kerberos kinit problems Error message: kinit: krb5_get_init_creds: Error from KDC: CLIENT EXPIRED Problem: Your Kerberos account has expired. Solution: Please see the User Accounts web page for more information on renewing your Kerberos account. Error message: kinit(v5): Cannot find KDC for requested realm while getting initial credentials Problem: \/etc\/krb5.conf file does not contain .FNAL.GOV information. Solutions: Replace \/etc\/krb5.conf with your OS-specific Fermilab-supplied version of…  More »<\/a><\/p>\n","protected":false},"author":16,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-556","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/comments?post=556"}],"version-history":[{"count":37,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/556\/revisions"}],"predecessor-version":[{"id":2053,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/556\/revisions\/2053"}],"wp:attachment":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/media?parent=556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}