Workers

The task manager offers a few different types of workers, internal, external and bare metal workers.

Depending on the type of task, it will be dispatched to a dedicated queue for the type of worker required.

External Workers (AWS Lambda)

These workers run on AWS Lambda outside of the VPC. They do not have direct access to the PostgreSQL database. They are typically used for tasks that communicate to external services for recon purposes. Unfortunately AWS Lambda does not support raw socket operations, so tasks that require raw socket access cannot be run on these workers. Examples of such tasks are TCP SYN scanning or custom TLS scanning.

Useful for creating distributed tasks towards services like crt.sh, Censys, Shodan, or other public services.

NOTE: they do NOT have access to the PostgreSQL database. This means storing task results are handled by passing SucceededWithData messages back to the Task Manager.

Internal Workers (AWS VPC bound Lambda)

These workers run on AWS Lambda inside of the VPC. They have read-write access to the tasks_data and recon tables in the PostgreSQL database. They are typically used for tasks that perform data mutations or filtering operations. Since they run inside of the VPC, they can connect directly to the PostgreSQL database.

Bare Metal Workers

Bare metal workers have raw socket access. This means they can use our custom network scanning engines like TcpSyn scanning and custom TLS scanning. There is a dedicated SQS queue for bare metal workers. On launch, the worker will poll SQS for new tasks to execute. Using environment variables, the worker can configure a timeout value to poll for messages on the SQS queue. If there are no messages within the given timeout, the worker will shutdown itself. This feature is useful for running the worker on EC2 to cut down costs.

When running on EC2, the workers are placed in the VPC containing the PostgreSQL database. This means the bare metal workers have read-write access to the tasks_data and recon tables in the PostgreSQL database. This makes them suitable for both scanning tasks and heavy duty data mutation tasks.

When running outside of AWS, the bare metal workers could use SucceededWithData messages to pass data back to the Task Manager. However, this is not implemented yet. It would require us to create either separate task definitions OR have a flag in the task definition to indicate if the worker is running inside or outside of AWS.

Ideas and Future Work

• Implement workers running on other environments like Azure, or bare metal hardware outside of AWS.