How to quickly level up your triaging and investigative skills at a SOC

It is no secret; SOC Analysts these days are inundated by an immense number of alerts that require their very own human attention or critical thinking. Efficiently and accurately going through these alerts to determine severity and whether or not an alert gets escalated to incident response can be a challenging and intimidating process for newcomers. Whether you are new to Cyber Security or recently joined a SOC, here are some basic tips and concepts to think about when approaching your alert triage and investigations.

Endpoint Protection

1. The capabilities of today’s endpoint protection platforms have come a long way, but it’s important to know they are not perfect and will not detect everything.  If your NGAV platform for instance detects, blocks, and cleans something, it’s still worth reviewing the surrounding telemetry to ensure all stages of a threat were detected and removed.  I’ve personally observed instances where only a part of an infection was blocked, while additional persistence mechanisms went undetected in Registry Run keys and Scheduled Tasks.
2. Just because a system is shown as Clean in the console does not make it necessarily true. Health Status should not be solely relied upon when making a final determination of whether a host was compromised.
3. Be skeptical of full AV scans for the same reason.  If a full scan completes and doesn’t find anything, that doesn’t mean the host actually is Clean, that just means the product was unable to detect what it was designed (or configured) to detect.
4. Depending on the maturity and capabilities of your org, perform additional types of forensics (host\memory\network) to help make a better determination.
   
   For additional information on training, resources, and tools in the various areas of digital forensics, visit this SANS link.

Log Visibility and SIEM

1. When it comes to logging in an aggregator or SIEM, if you don’t understand what’s being ingested, you won’t be able to make a good determination when investigating.  Spend a lot of time getting to know the available log sources in the product itself or through internal reports and internal documentation.
2. Hopefully you’re in a situation where your log visibility is mature and you generally have most of what you need to fulfill your day to day needs.  If your log visibility is poor, you’ll most likely need to pull logs directly from devices more often than not.  Don’t be afraid to engage other teams and SMEs to provide those logs to you and perhaps assist in the analysis.  If it makes sense to log that particular device in a SIEM, make a business case to take on that new log volume and get it ingested. That visibility will be readily available the next time you or another analyst requires it. Additional detections can be built from those new logs as well.
3. Know that you might be getting a log source in your SIEM, but a certain audit level or configuration on the OS or device side may limit what is being forwarded.  This is part of knowing what is being logged in your environment as previously mentioned, but it’s also important to spend time understanding how the various OSs work, what they log, and how they log it.
4. If you don’t know what portions of a raw log are being parsed out into searchable fields in a SIEM, it’s possible you might miss something in your investigation when searching for a value.  Spend time knowing how your tools work and what data is made to be searchable or indexed.  While they require more resources to perform, don’t be afraid to lean on wildcard type searches (Ex: Log Message contains “8.8.8[.]8”) from time to time until you get more comfortable with where metadata is being captured (Destination IP = “8.8.8[.]8”).

Google-Fu

1. Search engines like Google are one of the most valuable resources at the disposal of the Analyst to find Open Source Intel (OSINT) on the internet.  Leveraging different types of words and symbols, you can create more focused and complex searches to get hits on what you’re investigating.
   
   a) Double Quotes
   Search on whole command lines, parts of scripts, or strings using “double quotes” to see exact matches that otherwise would not pop up.
   
   b) AND, OR, and NOT
   Craft searches with multiple words and strings using AND, OR and NOT operators.
   
   c) Asterisk (*)
   Replace portions of your search with an Asterisk to perform wildcard searches.
   
   For additional search options, take a look at this.

Intel/Analyzer Sites

1. When indicators are shown as unknown or clean on intel/analyzer sites like VirusTotal, remain skeptical of the disposition as the indicator could still be malicious.  Further review multiple sites and sources (not just one) to determine whether the indicator was previously observed.  If you’re not able to track down enough information about a file for instance, consider analyzing it in a private sandbox like CuckooSandbox, for example.  It’s important to know that it’s common for the adversary to leverage custom malware designed for one specific environment they attack, so for an analyst to find little or no information about a new file or indicator is normal.
   
   Notable Intel/Analyzer sites:
   a) VirusTotal
   b) IBM X-Force Exchange
   c) AlientVault OTX

Searching

1. When it comes to alert triage, it’s common for new analysts to focus too much on what the alert puts in front of them. Usually it’s a small set of device metadata and event activity based on the detection criteria. In many cases, a review of the initial events can be enough to make your determination.  In other instances you’ll want to expand or narrow your search, and possibly pivot on certain data points in various ways.  Zooming out to review all logs on that system in a somewhat wider time range could be a next step in your investigation to examine surrounding activity leading up to the alert.  Depending on the detection, consider pivoting on certain event classifications like “Suspicious” or “Attack” to see if the same classification type was observed on other hosts around the same time. If the source of a potential attack shows an IP address, check to see if that IP address was seen in any other metadata fields like destination IP, as well as source IP.
2. When searching in general, avoid going down too many rabbit holes for long periods.  If you only have a 1 hour SLA to triage your alert and make a determination, be as efficient with your time as possible.  If it can take you 10 minutes to research how a strain of malware basically works, don’t spend the whole hour becoming an expert on it when you could be helping tell the story of how it got into the environment and what was impacted.  You can always continue your research at a later time.
   
   
   ____________
   
   Any other advice you can share? Have any questions? Let’s continue the conversation either here or on Twitter @CyberCovfefe.