Posted inSlack Watchman Tools

Slack Watchman – Version 3.0.0

September 4, 2020

In what should be the last major version bump for a while, today I release Slack Watchman 3.0.0. This update adds some transformational features that help move Slack Watchman into the realms of being a truly useful solution for enterprise.

Slack Watchman is available on GitHub: https://github.com/PaperMtn/slack-watchman

Or can be installed from PyPI: pip install slack-watchman

Slack Watchman Rules!

The first big update is the introduction of rules based searching. Previously, Slack Watchman operated by hard coded search terms and regex patterns, and adding new searches required updates to the code.

Now it uses YAML formatted rules to provide all of the information needed to search. This is what a Slack Watchman rule looks like:

---
filename: github_api_tokens.yaml
enabled: true
meta:
  name: GitHub API Tokens
  author: PaperMtn
  date: '2020-08-14'
  description: Detects exposed GitHub API OAuth tokens
  severity: '90'
category: tokens
scope:
- messages
file_types:
test_cases:
  match_cases:
  - 'https://github.com/login/oauth/$access_token=abcabc123&token_type=bearer'
  fail_cases:
  - 'GET https://github.com/login/oauth/authorize'
strings:
- api.github.com
- github.com/login/oauth/
- github access_token
pattern: '[0-9a-zA-Z]{20,40}'

Using YAML rules gives these advantages:

Added metadata

Additional information can be added to each search that is useful in dealing with the detection, such as severity. This is particularly useful when ingesting data into a SIEM (more on that later)

Enable/disable rules

Don’t care about Office docs? or certificate files? You can disable these rules to stop them running. This means you can search your Slack workspace efficiently, and only get the results you want.

Testing regex

Tests cases are included in the rules, which test the effectiveness of the regex you supply against strings that should match, and those that don’t. Many projects supply you with regex patterns and you have to trust they are effective, Slack Watchman now lets you confirm it.

Really easy to new detections

These YAML formatted rules have been designed to be as easy as possible for humans to read, and to create. Adding a search is now incredibly simple. Just create a rule and put it in the rules directory, Slack Watchman will then search for it on the next run

Adding your own rules

There is documentation on how to format rules in the GitHub repository, so I won’t go over that here.

Once you have written your own rules, you can build Slack Watchman from source to use them.

Logging

The other big update is new logging options for Slack Watchman. CSV is great for humans to review, but to automate actions based on the output, something more was required. Now, alongside the default logging to CSV, you have the option to log output to:

  • Log file
  • Stdout
  • TCP stream

These new logging options output logs in JSON format. They look like this:

{"localtime": "2020-00-00 00:00:00,000", "level": "NOTIFY", "source": "Slack Watchman", "workspace": "Westeros Inc", "scope": "messages", "type": "Twitter API Tokens", "severity": "90", "detection": {"channel_name": "lannister_chat", "message_id": "abc123", "permalink": "https://...", "posted_by": "tywin.lannister", "text": "<https://api.twitter.com/oauth/authorize?oauth_token=xXxXxX>", "timestamp": "2020-00-00 00:00:00"}}

SIEM/Log analyser ingestion

The main benefit of these new logging options is that they provide the data in a JSON format ready to be ingested in a SIEM.

If you configure Slack Watchman to run regularly and ingest these logs into your SIEM, you can automate a lot of the work to generate alerts, and even automate remediating any issues detected.

File logging

File logging saves JSON formatted logs to a file.

The path where you want to output the file needs to be passed when running Slack Watchman. This can be done via the .conf file:

slack_watchman:
  token: xoxp-xxxxxxxx
  logging:
    file_logging:
      path: /var/put_my_logs_here/
    json_tcp:
      host:
      port:

Or by setting your log path in the environment variable: SLACK_WATCHMAN_LOG_PATH

If file logging is selected as the output option but no path is given, Slack Watchman defaults to your home directory.

The filename will be slack_watchman.log

Note: Slack Watchman does not handle the rotation of the file. You would need a solution such as logrotate for this.

Stdout logging

Stdout logging sends JSON formatted logs to Stdout, for you to capture however you want:

{"localtime": "2020-09-04 20:27:10,903", "level": "INFO", "source": "Slack Watchman", "message": "Slack Watchman started execution"}
{"localtime": "2020-09-04 20:27:10,903", "level": "INFO", "source": "Slack Watchman", "message": "Searching workspace: Westeros Inc"}
{"localtime": "2020-09-04 20:27:11,137", "level": "INFO", "source": "Slack Watchman", "message": "Workspace URL: https://westerosinc.slack.com/"}
{"localtime": "2020-09-04 20:27:11,137", "level": "INFO", "source": "Slack Watchman", "message": "Importing rules..."}
{"localtime": "2020-09-04 20:27:11,331", "level": "INFO", "source": "Slack Watchman", "message": "38 rules loaded"}
{"localtime": "2020-09-04 20:27:11,331", "level": "INFO", "source": "Slack Watchman", "message": "Searching tokens"}
{"localtime": "2020-09-04 20:27:11,331", "level": "INFO", "source": "Slack Watchman", "message": "Searching for posts containing Twitter API Tokens"}
...

All output is logged to Stdout in JSON format so it can be managed easily. (No ASCII art with this option I’m afraid)

TCP stream logging

With this option, JSON formatted logs are sent to a destination of your choosing via TCP

You will need to pass Slack Watchman a host and port to receive the logs, either via .conf file:

slack_watchman:
  token: xoxp-xxxxxxxx
  logging:
    file_logging:
      path:
    json_tcp:
      host: localhost
      port: 9020

Or by setting the environment variables SLACK_WATCHMAN_HOST and SLACK_WATCHMAN_PORT

Deduplication

All logging methods now deduplicate output before writing them.

Other Improvements

The whole codebase has gone through a refactor, as is often the way when you revisit a project after some time away. Slack Watchman now makes more efficient use of rate limit checking to make sure you only back off when you need to.

In general the code is now a lot more ‘grown up’. Classes have been implemented to reduce duplication. In general, Slack Watchman should now run more efficiently

What you need to do

If you have been running Slack Watchman previously, there are a few things to bear in mind:

  • This version changes the format of the .conf file. If you are upgrading and previously used a .conf file to provide the Slack token, make sure you have read the instructions on how to reformat the file.
  • Custom searching by text file is now deprecated. If you wish to use this feature, create your own rule with your custom strings and the category custom. See the rules documentation for more information.

Future Plans

Rule based searching makes Slack Watchman really easy to add to and to customise to your own needs. I plan on adding more rules (as well as features), but I would love it if others contributed rules as well. There should be enough information on GitHub to help you with rule creation, but feel free to raise an issue, or even contact me on Twitter, if you want to talk about it. Please do create a pull request to add some of your rules back into Slack Watchman.

Going forward, while I will still be working on Slack Watchman, I will be turning the Watchman’s gaze towards other systems as well, so keep an eye out for that.

As always, feel free to raise any issues via GitHub, or ping me directly on Twitter: https://twitter.com/_PaperMtn

Last updated on September 4, 2020