Troubleshooting

Osquery (Mac OS X)

No node key

There are a number of reasons for this error:

  • Your enrollment secret is invalid. Are you re-installing Zercurity from another account or deployment server? Try removing the enroll.dat file within the Zercurity installation folder.
  • Zercurity has either blocked an enrollment from talking place. Please contact our support team to help resolve this for you.
Failed enrollment request to https://api.zercurity.com/v1/osquery/enroll (No node key returned from TLS enroll plugin) retrying...

Accessing logs

tail -f /var/log/zercurity_osqueryd_stderr.log

Restarting Osquery

sudo /bin/launchctl unload "/Library/LaunchDaemons/com.zercurity.osqueryd.plist"
sudo /bin/launchctl load "/Library/LaunchDaemons/com.zercurity.osqueryd.plist"

Watchdog Issues (Performance)

By default Zercurity has configured Osquery to limit the resources used by the agent. Sometimes you may want to use more complex and expensive queries that use more system resources. Obviously this will impact the users system performance.

Below are the Default Zercurity settings:

--watchdog_level=0  (Performance limit level (0=normal, 1=restrictive, -1=off))
--watchdog_memory_limit=400  (Override watchdog profile memory limit (e.g., 300, for 300MB))
--watchdog_utilization_limit=21  (Override watchdog profile CPU utilization limit)

If you’ve run a query and received the following error, you will need to tweak the Osquery watchdog configuration:

osqueryd worker (1184) stopping: Maximum sustainable CPU utilization limit exceeded: 12

You can either override the settings on the Zercurity installer via Settings -> Configuration. Or you can modify these settings locally. The Osquery flags file is stored in the following locations.

Mac OSX

sudo vim /Library/LaunchDaemons/com.zercurity.osqueryd.plist

Windows

C:\ProgramData\zercurity\osquery\osquery.flags

Linux

sudo vim /etc/osquery/osquery.flags

Warning

Once you have updated the osquery.flags file you will either need to reboot your system or restart the Osquery daemon as per the steps above in order for these changes to take effect.

Google Santa

Zercurity makes use of Google Santa for its Mac OSX application whitelisting and blacklisting.

Force sync rules

Sometimes you may want to forcefully synchronise your local machine with the remote server to ensure all rules are up-to-date. The sync command will also send any pending updates back to the Zercurity server.

sudo santactl sync

Santa daemon failing to start

If you’ve installed Zercurity locally without using an MDM you will most likely encounter the following error. As during the installation you may not have given permission to Santa to install its System extension.

sudo santactl sync
An error occurred communicating with the daemon, is it running?

The first thing to do is check the logs as described below.

Accessing logs

It maybe that you need to manually “Allow” the Santa system extension. The installer should have prompted you to do this if your not using an MDM.

SystemExtension authorisation

If that still doesn’t work then make sure that Santa has “Full Disk Access”. You may need to reload the Santa daemon after completing this step.

Full Disk Access

Accessing logs

You can show the logs for Google Santa using the following command.

/usr/bin/log show --info --debug --predicate 'senderImagePath CONTAINS "Santa"' --last 1d

SystemExtension authorisation

If you see the following error then either the user needs to accept permission for Santa to run under System->Security&Privacy or the Zercurity MDM needs to be applied to the system via the MDM manager i.e. JAMF.

Santa: I Santa: SystemExtension "com.google.santa.daemon" request needs user approval
../_images/installer_osx_error_03.png

If you don’t see the option to accept permission under “System->Security & Privacy” to run Santa. Try closing the Security window and run the following command which should force the prompt.

Warning

Sometimes the “Allow” menu button may not show until you’ve closed the window and revisited it.

sudo /bin/launchctl unload -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist"
/Applications/Santa.app/Contents/MacOS/Santa --load-system-extension
sudo /bin/launchctl load -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist"

Full Disk Access

Google’s Santa requires full disk access in order for the Daemon to start.

../_images/trouble_osx_disk_access_full.png

Once checked you’ll need to reload the service.

sudo /bin/launchctl unload -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist"
sudo /bin/launchctl load -w "/Library/LaunchDaemons/com.google.santa.bundleservice.plist"

Missing Zercurity profile

The Zercurity installer should automatically install our profile. If the profile can’t be found or was removed you’ll get the following error when trying to manually sync Santa using santactl.

sudo santactl sync --debug
Missing SyncBaseURL. Exiting.

To resolve this you’ll need to re-install Zercurity.

Bad certificates file

In the event that your /usr/local/zercurity/zercurity.pem is missing or incomplete. You’ll receive the following error:

sudo santactl sync --debug
Server Trust: Server isn't trusted. SecTrustResultType: 5
Unable to verify server identity.
santactl: E santactl: HTTP Response: -999 cancelled
santactl: E santactl: Preflight failed, will try again once https://api.zercurity.local/v1/santa/ is reachable

Santa has failed to connect to the remote server using the provided SSL bundle. The PEM file must contain the complete certificate chain. Including both the Certificate authority (CA) and its intermediate certificates.

You can validate your PEM file using the following command:

openssl verify -CAfile /usr/local/zercurity/zercurity.pem \
  -untrusted /usr/local/zercurity/zercurity.pem \
  /usr/local/zercurity/zercurity.pem

All certificates must be included in the same file. The -untrusted flag is used to include the intermediate certificate. These are the certificates between the CA and server certificate.

If the openssl verify returns /usr/local/zercurity/zercurity.pem: OK and you are still receiving the -999 error then its likely your certificate does not conform to Apple’s transport security (ATS) requirements.

  • TLS server certificates and issuing CAs using RSA keys must use key sizes greater than or equal to 2048 bits. Certificates using RSA key sizes smaller than 2048 bits are no longer trusted for TLS.
  • TLS server certificates and issuing CAs must use a hash algorithm from the SHA-2 family in the signature algorithm. SHA-1 signed certificates are no longer trusted for TLS.
  • TLS server certificates must present the DNS name of the server in the Subject Alternative Name extension of the certificate. DNS names in the CommonName of a certificate are no longer trusted.

https://support.apple.com/en-us/HT210176

You can check that a certificate has the correct extensions with:

openssl x509 -in api.local.zercurity.com.crt -text -noout

Lastly, if you receive the following error:

HTTP Response: -1200 An SSL error has occurred and a secure connection to the server cannot be made.
Preflight failed, aborting run

The zercurity CA needs to be added to your system roots trust-store. This can be done using the following command:

sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain /usr/local/zercurity/zercurity.pem