.. include:: /includes.txt =============== Troubleshooting =============== Osquery ======= No node key ----------- There are a number of reasons for this error: * Your enrollment secret is invalid. Are you re-installing Zercurity from another account or deployment server? Try removing the ``enroll.dat`` file within the Zercurity installation folder. * Zercurity has either blocked an enrollment from talking place. Please contact our support team to help resolve this for you. .. code-block:: bash Failed enrollment request to https://api.zercurity.com/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying... Accessing logs -------------- **Mac OSX** .. code-block:: bash sudo tail -f /var/log/zercurity_osqueryd_stderr.log **Windows** .. code-block:: powershell C:\ProgramData\zercurity\log **Linux** .. code-block:: bash sudo tail -f /var/log/syslog Restarting Osquery ------------------ .. code-block:: bash sudo /bin/launchctl unload "/Library/LaunchDaemons/com.zercurity.osqueryd.plist" sudo /bin/launchctl load "/Library/LaunchDaemons/com.zercurity.osqueryd.plist" Watchdog Issues (Performance) ----------------------------- By default Zercurity has configured Osquery to limit the resources used by the agent. Sometimes you may want to use more complex and expensive queries that use more system resources. Obviously this will impact the users system performance. Below are the Default Zercurity settings: .. code-block:: bash --watchdog_level=0 (Performance limit level (0=normal, 1=restrictive, -1=off)) --watchdog_memory_limit=400 (Override watchdog profile memory limit (e.g., 300, for 300MB)) --watchdog_utilization_limit=21 (Override watchdog profile CPU utilisation limit) If you've run a query and received the following error, you will need to tweak the Osquery watchdog configuration: .. code-block:: bash osqueryd worker (1184) stopping: Maximum sustainable CPU utilisation limit exceeded: 12 You can either override the settings on the Zercurity installer via **Settings -> Configuration**. Or you can modify these settings locally. The Osquery flags file is stored in the following locations. **Mac OSX** .. code-block:: bash sudo vim /Library/LaunchDaemons/com.zercurity.osqueryd.plist **Windows** .. code-block:: powershell C:\ProgramData\zercurity\osquery\osquery.flags **Linux** .. code-block:: bash sudo vim /etc/osquery/osquery.flags .. warning:: Once you have updated the `osquery.flags` file you will either need to reboot your system or restart the Osquery daemon as per the steps above in order for these changes to take effect. Google Santa ============ Zercurity makes use of Google Santa for its Mac OSX application whitelisting and blacklisting. Force sync rules ---------------- Sometimes you may want to forcefully synchronise your local machine with the remote server to ensure all rules are up-to-date. The ``sync`` command will also send any pending updates back to the Zercurity server. .. code-block:: bash sudo santactl sync Santa daemon failing to start ----------------------------- If you've installed Zercurity locally without using an MDM you will most likely encounter the following error. As during the installation you may not have given permission to Santa to install its System extension. .. code-block:: bash sudo santactl sync An error occurred communicating with the daemon, is it running? The first thing to do is check the logs as described below. :ref:`santa_logs` It maybe that you need to manually **"Allow"** the Santa system extension. The installer should have prompted you to do this if your not using an MDM. .. code-block:: bash sudo systemextensionsctl list :ref:`santa_extension` If that still doesn't work then make sure that Santa has "Full Disk Access". You may need to reload the Santa daemon after completing this step. :ref:`santa_full_disk_access` .. _santa_logs: Accessing logs -------------- You can show the logs for Google Santa using the following command. .. code-block:: bash /usr/bin/log show --info --debug --predicate 'senderImagePath CONTAINS "Santa"' --last 1d .. _santa_extension: SystemExtension authorisation ----------------------------- If you see the following error then either the user needs to accept permission for Santa to run under **System->Security&Privacy** or the Zercurity MDM needs to be applied to the system via the MDM manager i.e. JAMF. .. code-block:: bash Santa: I Santa: SystemExtension "com.google.santa.daemon" request needs user approval .. image:: /_static/installer_osx_error_03.png If you don't see the option to accept permission under **"System->Security & Privacy"** to run Santa. Try closing the Security window and run the following command which should force the prompt. .. warning:: Sometimes the "Allow" menu button may not show until you've closed the window and revisited it. .. code-block:: bash sudo /bin/launchctl unload -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist" /Applications/Santa.app/Contents/MacOS/Santa --load-system-extension sudo /bin/launchctl load -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist" .. _santa_full_disk_access: Full Disk Access ---------------- Google's Santa requires full disk access in order for the Daemon to start. .. image:: /_static/trouble_osx_disk_access_full.png Once checked you'll need to reload the service. .. code-block:: bash sudo /bin/launchctl unload -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist" sudo /bin/launchctl load -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist" .. _santa_profile: Missing Zercurity profile ------------------------- The Zercurity installer should automatically install our profile. If the profile can't be found or was removed you'll get the following error when trying to manually ``sync`` Santa using ``santactl``. .. code-block:: bash sudo santactl sync --debug Missing SyncBaseURL. Exiting. Firstly, via the System Preferences Panel. Click on Profiles and check that either the newer "Zercurity Google Santa" profile is installed. Or the "com.google.santa" profile is installed. If the profile doesn't exist you can either try to re-install Zercurity. Or manually install the Google Santa configuration profile like so: **Mac OSX** .. code-block:: bash /usr/bin/profiles -I -F "/usr/local/zercurity/santa/com.google.santa.mobileconfig" **Mac OSX BigSur (> 10.15)** .. code-block:: bash open /System/Library/PreferencePanes/Profiles.prefPane "/usr/local/zercurity/santa/com.google.santa.mobileconfig" If the following commands still don't work then try restarting the host machine. If you're still running into trouble, then please get in touch. Debugging Errors ---------------- On some rare instances where you need to troubleshoot why your asset is failing to download the latest rules or enroll correctly. For example: .. code-block:: bash sudo santactl sync --debug Rule download failed, aborting run You can manually make the request yourself to see what errors your Zercurity might be returning. In most cases this might be a name resolution error, proxy or other miss configuration. .. code-block:: bash curl -vvv -X POST https://api.zercurity.com/v1/santa/ruledownload/`sudo cat /usr/local/zercurity/enroll.dat` Bad certificates file --------------------- In the event that your ``/usr/local/zercurity/zercurity.pem`` is missing or incomplete. You'll receive the following error: .. code-block:: bash sudo santactl sync --debug Server Trust: Server isn't trusted. SecTrustResultType: 5 Unable to verify server identity. santactl: E santactl: HTTP Response: -999 cancelled santactl: E santactl: Preflight failed, will try again once https://api.zercurity.local/v1/santa/ is reachable Santa has failed to connect to the remote server using the provided SSL bundle. The PEM file must contain the complete certificate chain. Including both the Certificate authority (CA) and its intermediate certificates. You can validate your PEM file using the following command: .. code-block:: bash openssl verify -CAfile /usr/local/zercurity/zercurity.pem \ -untrusted /usr/local/zercurity/zercurity.pem \ /usr/local/zercurity/zercurity.pem All certificates must be included in the same file. The ``-untrusted`` flag is used to include the intermediate certificate. These are the certificates between the CA and server certificate. If the ``openssl verify`` returns ``/usr/local/zercurity/zercurity.pem: OK`` and you are still receiving the ``-999`` error then its likely your certificate does not conform to Apple's transport security (ATS) requirements. - TLS server certificates and issuing CAs using RSA keys must use key sizes greater than or equal to 2048 bits. Certificates using RSA key sizes smaller than 2048 bits are no longer trusted for TLS. - TLS server certificates and issuing CAs must use a hash algorithm from the SHA-2 family in the signature algorithm. SHA-1 signed certificates are no longer trusted for TLS. - TLS server certificates must present the DNS name of the server in the Subject Alternative Name extension of the certificate. DNS names in the CommonName of a certificate are no longer trusted. https://support.apple.com/en-us/HT210176 You can check that a certificate has the correct extensions with: .. code-block:: bash openssl x509 -in api.local.zercurity.com.crt -text -noout Lastly, if you receive the following error: .. code-block:: bash HTTP Response: -1200 An SSL error has occurred and a secure connection to the server cannot be made. Preflight failed, aborting run The zercurity CA needs to be added to your system roots trust-store. This can be done using the following command: .. code-block:: bash sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /usr/local/zercurity/zercurity.pem