Crawler implementation.
Crawler
(config)[source]¶Bases: google.cloud.forseti.services.inventory.base.crawler.Crawler
Simple single-threaded Crawler implementation.
dispatch
(callback)[source]¶Dispatch crawling of a subtree.
Parameters: | callback (function) – Callback to dispatch. |
---|
on_child_error
(error)[source]¶Process the error generated by child of a resource
Inventory does not stop for children errors but raise a warning
Parameters: | error (str) – error message to handle |
---|
run
(resource)[source]¶Run the crawler, given a start resource.
Parameters: | resource (object) – Resource to start with. |
---|---|
Returns: | The filled progresser described in inventory |
Return type: | QueueProgresser |
update
(resource)[source]¶Update the row of an existing resource
Parameters: | resource (Resource) – Resource to update. |
---|---|
Raises: | Exception – Reraises any exception. |
CrawlerConfig
(storage, progresser, api_client, variables=None)[source]¶Bases: google.cloud.forseti.services.inventory.base.crawler.CrawlerConfig
Crawler configuration to inject dependencies.
ParallelCrawler
(config)[source]¶Bases: google.cloud.forseti.services.inventory.crawler.Crawler
Multi-threaded Crawler implementation.
dispatch
(callback)[source]¶Dispatch crawling of a subtree.
Parameters: | callback (function) – Callback to dispatch. |
---|
on_child_error
(error)[source]¶Process the error generated by child of a resource
Inventory does not stop for children errors but raise a warning
Parameters: | error (str) – error message to handle |
---|
run
(resource)[source]¶Run the crawler, given a start resource.
Parameters: | resource (Resource) – Resource to start with. |
---|---|
Returns: | The filled progresser described in inventory |
Return type: | QueueProgresser |
ParallelCrawlerConfig
(storage, progresser, api_client, threads=10, variables=None)[source]¶Bases: google.cloud.forseti.services.inventory.base.crawler.CrawlerConfig
Multithreaded crawler configuration, to inject dependencies.
_api_client_factory
(storage, config, parallel)[source]¶Creates the proper initialized API client based on the configuration.
Parameters: |
|
---|---|
Returns: | The initialized api client implementation class. |
Return type: |
_crawler_factory
(storage, progresser, client, parallel)[source]¶Creates the proper initialized crawler based on the configuration.
Parameters: |
|
---|---|
Returns: | The initialized crawler implementation class. |
Return type: | Union[Crawler, ParallelCrawler] |
_root_resource_factory
(config, client)[source]¶Creates the proper initialized crawler based on the configuration.
Parameters: |
|
---|---|
Returns: | The initialized root resource. |
Return type: |
run_crawler
(storage, progresser, config, parallel=True)[source]¶Run the crawler with a determined configuration.
Parameters: |
|
---|---|
Returns: | The progresser implemented in inventory |
Return type: |