Stack Overflow Watchman

After a break, where my attention has been focused elsewhere, I’ve been drawn back to one of my first loves, exposed secrets detection. The result is a brand new app in the Watchman family; Stack Overflow Watchman.

Features

If you’ve used any of the other applications in the Watchman family, you will know what to expect. Stack Overflow Watchman brings all of the standard Watchman features to the catchily named Stack Overflow Enterprise for Teams.

It makes use of the centralised Watchman Signatures repository to stay up to date with the latest signatures.

You can also select how far back to search from:

• All time
• Last month
• Last week
• Last 24 hours

Logging

There are options for logging: stdout, or logging to JSON, which is perfect for ingesting into a SIEM.

JSON logging is more verbose, with more fields for each finding. This is an example JSON finding:

{
  "timestamp": "2024-05-06 13:53:15,645",
  "level": "NOTIFY",
  "scope": "question",
  "severity": "90",
  "detection_type": "Alibaba IAM Secret Access Key",
  "detection_data": {
    "match_string": "HERESTHETOKEN",
    "match_type": "question",
    "question": {
      "accepted_answer_id": null,
      "answer_count": 1,
      "bounty_amount": null,
      "bounty_closes_date": null,
      "closed_date": null,
      "closed_reason": null,
      "collectives": null,
      "community_owned_date": null,
      "content_license": null,
      "creation_date": "2022-07-22T13:46:58.000000",
      "is_answered": true,
      "last_activity_date": "2023-08-19T10:00:00.000000",
      "last_edit_date": "2022-07-25T08:40:34.000000",
      "link": "https://papermtn.stackenterprise.co/questions/1973",
      "locked_date": null,
      "migrated_from": null,
      "migrated_to": null,
      "owner": {
        "account_id": 53,
        "display_name": "PaperMtn",
        "link": "https://papermtn.stackenterprise.co/users/53/",
        "profile_image": "https://www.gravatar.com/avatar/",
        "reputation": 33,
        "user_id": 53,
        "user_type": "registered"
      },
      "posted_by_collectives": null,
      "protected_date": null,
      "question_id": 1973,
      "score": 0,
      "tags": [
        "test"
      ],
      "title": "My test question",
      "view_count": 65
    }
  }
}

Stdout logging is more slimmed down, giving you the key information.

Requirements

Stack Overflow Version

Stack Overflow Watchman requires you to have a
Stack Overflow Enterprise for Teams licence (it won’t work with the free version, as this doesn’t provide API functionality).

API Token

You will need to generate an API token to authenticate to Stack Overflow, to do this, log into your account and under your profile go to Settings -> API applications.

You can pass this API token to Stack Overflow Watchman using the environment variable STACK_OVERFLOW_WATCHMAN_TOKEN

Domain

Finally, you need to give the domain for your site. Domains usually end in .stackoverflow.co. If your site is papermtn.stackoverflow.co, then give the domain papermtn.

Pass this with the environment variable STACK_OVERFLOW_WATCHMAN_TOKEN

Using Stack Overflow Watchman

Installing

You can install Stack Overflow Watchman from PyPI using pip:

python3 -m pip install stack-overflow-watchman

Or, there is also a Docker image available:

docker pull papermountain/stack-overflow-watchman:latest

Running

Avaiable options can be found by running stack-overflow-watchman -h, if you wanted to run a search over all time, defaulting to stdout, use the command:

stack-overflow-watchman --timeframe a