google.cloud.forseti.services.inventory.crawler module

Crawler implementation.

class Crawler(config)[source]

Bases: google.cloud.forseti.services.inventory.base.crawler.Crawler

Simple single-threaded Crawler implementation.

dispatch(callback)[source]

Dispatch crawling of a subtree.

Parameters:callback (function) – Callback to dispatch.
get_client()[source]

Get the GCP API client.

Returns:GCP API client
Return type:object
on_child_error(resource_full_name, error)[source]

Process the error generated by child of a resource

Inventory does not stop for children errors but raise a warning
Parameters:
  • resource_full_name (str) – The full name of the resource that raised the error.
  • error (str) – error message to handle
run(resource)[source]

Run the crawler, given a start resource.

Parameters:resource (object) – Resource to start with.
Returns:The filled progresser described in inventory
Return type:QueueProgresser
visit(resource)[source]

Handle a newly found resource.

Parameters:resource (object) – Resource to handle.
Raises:Exception – Reraises any exception.
write(resource)[source]

Save resource to storage.

Parameters:resource (object) – Resource to handle.
class CrawlerConfig(storage, progresser, api_client, variables=None)[source]

Bases: google.cloud.forseti.services.inventory.base.crawler.CrawlerConfig

Crawler configuration to inject dependencies.

class ParallelCrawler(config)[source]

Bases: google.cloud.forseti.services.inventory.crawler.Crawler

Multi-threaded Crawler implementation.

_process_queue()[source]

Process items in the queue until the shutdown event is set.

_start_workers()[source]

Start a pool of worker threads for processing the dispatch queue.

dispatch(callback)[source]

Dispatch crawling of a subtree.

Parameters:callback (function) – Callback to dispatch.
run(resource)[source]

Run the crawler, given a start resource.

Parameters:resource (Resource) – Resource to start with.
Returns:The filled progresser described in inventory
Return type:QueueProgresser
class ParallelCrawlerConfig(storage, progresser, api_client, threads, variables=None)[source]

Bases: google.cloud.forseti.services.inventory.base.crawler.CrawlerConfig

Multithreaded crawler configuration, to inject dependencies.

_api_client_factory(config, threads, inventory_index_id)[source]

Creates the proper initialized API client based on the configuration.

Parameters:
  • config (object) – Inventory configuration on server.
  • threads (int) – how many threads to use.
  • inventory_index_id (int) – The inventory index ID for this export.
Returns:

The initialized api client implementation class.

Return type:

Union[gcp.ApiClientImpl, cai_gcp_client.CaiApiClientImpl]

_crawler_factory(storage, progresser, client, parallel, threads)[source]

Creates the proper initialized crawler based on the configuration.

Parameters:
  • storage (object) – Storage implementation to use.
  • progresser (object) – Progresser to notify status updates.
  • client (object) – The API client instance.
  • parallel (bool) – If true, use the parallel crawler implementation.
  • threads (int) – how many threads to use when running in parallel
Returns:

The initialized crawler implementation class.

Return type:

Union[Crawler, ParallelCrawler]

_root_resource_factory(config, client)[source]

Creates the proper initialized crawler based on the configuration.

Parameters:
  • config (object) – Inventory configuration on server.
  • client (object) – The API client instance.
Returns:

The initialized root resource.

Return type:

Resource

run_crawler(storage, progresser, config, parallel=True, threads=10)[source]

Run the crawler with a determined configuration.

Parameters:
  • storage (object) – Storage implementation to use.
  • progresser (object) – Progresser to notify status updates.
  • config (object) – Inventory configuration on server.
  • parallel (bool) – If true, use the parallel crawler implementation.
  • threads (int) – how many threads to use when running in parallel.
Returns:

The progresser implemented in inventory

Return type:

QueueProgresser