google.cloud.forseti.services.inventory.base.cloudasset module

Forseti Inventory Cloud Asset API integration.

exception StreamError[source]

Bases: Exception

Raised for errors streaming results from GCS to local DB.

_clear_cai_data(engine)[source]

Clear CAI data from storage.

Parameters:engine (object) – Database engine.
_download_cloudasset_data(config, inventory_index_id)[source]

Download cloud asset data.

Parameters:
  • config (InventoryConfig) – Inventory config.
  • inventory_index_id (int) – The inventory index ID for this export.
Yields:

str – GCS path of the cloud asset file.

_export_assets(cloudasset_client, config, root_id, content_type, inventory_index_id)[source]

Worker function for exporting assets and downloading dump from GCS.

Parameters:
  • cloudasset_client (CloudAssetClient) – CloudAsset API client interface.
  • config (object) – Inventory configuration on server.
  • root_id (str) – The name of the parent resource to export assests under.
  • content_type (ContentTypes) – The content type to export.
  • inventory_index_id (int) – The inventory index ID for this export.
Returns:

The path to the GCS object created by the CloudAsset API.

Return type:

str

Raises:

ValueError – Raised if the server configuration for CAI export is invalid.

_get_gcs_path(base_path, content_type, root_id, inventory_index_id)[source]

Generate a GCS object path for CAI dump.

Parameters:
  • base_path (str) – The GCS bucket, starting with ‘gs://’.
  • content_type (str) – The Cloud Asset content type for this export.
  • root_id (str) – The root resource ID for this export.
  • inventory_index_id (int) – The inventory index ID for this export.
Returns:

The full path to a GCS object to store export the data to.

Return type:

str

_stream_cloudasset_worker(cai_data, engine, output_queue)[source]

Worker to stream data from GCS into sqlite temporary table.

Parameters:
  • cai_data (file) – An open file like pipe.
  • engine (sqlalchemy.engine.Engine) – Database engine to write data to.
  • output_queue (collections.deque) – A queue storing the results of this thread.
_stream_gcs_to_database(gcs_object, engine, storage_client)[source]

Stream data from GCS into a local database using pipes.

Parameters:
  • gcs_object (str) – The full path to the GCS object to read.
  • engine (sqlalchemy.engine.Engine) – The db engine to store the data in.
  • storage_client (storage.StorageClient) – The storage client to use to download data from GCS.
Returns:

The number of rows stored in the database.

Return type:

int

Raises:

StreamError – Raised on any errors streaming data from GCS.

load_cloudasset_data(engine, config, inventory_index_id)[source]

Export asset data from Cloud Asset API and load into storage.

Parameters:
  • engine (object) – Database engine.
  • config (InventoryConfig) – Inventory configuration on server.
  • inventory_index_id (int) – The inventory index ID for this export.
Returns:

The count of assets imported into the database, or None if there

is an error.

Return type:

int