google.cloud.forseti.services.inventory.storage module

Inventory storage implementation.

class BufferedDbWriter(session, max_size=1024, max_packet_size=25165824.0, commit_on_flush=False)[source]

Bases: object

Buffered db writing.

add(obj, estimated_length=0)[source]

Add an object to the buffer to write to db.

Parameters:
  • obj (object) – Object to write to db.
  • estimated_length (int) – The estimated length of this object.
flush()[source]

Flush all pending objects to the database.

class CaiDataAccess[source]

Bases: object

Access to the CAI temporary store table.

static clear_cai_data(session)[source]

Deletes all temporary CAI data from the cai temporary table.

Parameters:session (object) – Database session.
Returns:The number of rows deleted.
Return type:int
static fetch_cai_asset(content_type, asset_type, name, session)[source]

Returns a single resource from the cai temporary store.

Retries query on exception up to 5 times.

Parameters:
  • content_type (ContentTypes) – The content type to return.
  • asset_type (str) – The asset type to return.
  • name (str) – The resource to return.
  • session (object) – Database session.
Returns:

The content data for the specified resource.

Return type:

dict

static iter_cai_assets(content_type, asset_type, parent_name, session)[source]

Iterate the objects in the cai temporary table.

Retries query on exception up to 5 times.

Parameters:
  • content_type (ContentTypes) – The content type to return.
  • asset_type (str) – The asset type to return.
  • parent_name (str) – The parent resource to iter children under.
  • session (object) – Database session.
Yields:

object – The content_type data for each resource.

static populate_cai_data(data, session)[source]

Add assets from cai data dump into cai temporary table.

Parameters:
  • data (file) – A file like object, line delimeted text dump of json data representing assets from Cloud Asset Inventory exportAssets API.
  • session (object) – Database session.
Returns:

The number of rows inserted

Return type:

int

class CaiTemporaryStore(name, parent_name, content_type, asset_type, asset_data)[source]

Bases: object

CAI temporary inventory table.

UNPARENTED_ASSETS = frozenset({'cloudbilling.googleapis.com/BillingAccount', 'cloudresourcemanager.googleapis.com/Organization'})
static _get_parent_name(asset)[source]

Determines the parent name from the resource data.

Parameters:asset (dict) – An Asset object.
Returns:The parent name for the resource.
Return type:str
asset_data = None
asset_type = None
content_type = None
classmethod delete_all(session)[source]

Deletes all rows from this table.

Parameters:session (object) – db session
Returns:The number of rows deleted.
Return type:int
Raises:Exception – Reraises any exception.
extract_asset_data(content_type)[source]

Extracts the data from the asset protobuf based on the content type.

Parameters:content_type (ContentTypes) – The content type data to extract.
Returns:
The dict representation of the asset
data and an Asset metadata along with it.
Return type:Tuple[dict, AssetMetadata]
classmethod from_json(asset_json)[source]

Creates a database row object from the json data in a dump file.

Parameters:asset_json (str) – The json representation of an Asset.
Returns:database row object or None if there is no data.
Return type:object
classmethod initialize(metadata, collation='utf8_bin')[source]

Create the table schema based on run time arguments.

Used to fix the column collation value for non-MySQL database engines.

Parameters:
  • metadata (object) – The sqlalchemy MetaData to associate the table with.
  • collation (str) – The collation value to use.
name = None
parent_name = None
class Categories[source]

Bases: enum.Enum

Inventory Categories.

billing_info = 5
dataset_policy = 4
enabled_apis = 6
gcs_policy = 3
iam_policy = 2
kubernetes_service_config = 7
resource = 1
class ContentTypes[source]

Bases: enum.Enum

Cloud Asset Inventory Content Types.

iam_policy = 2
resource = 1
class DataAccess[source]

Bases: object

Access to inventory for services.

classmethod delete(session, inventory_index_id)[source]

Delete an inventory index entry by id.

Parameters:
  • session (object) – Database session.
  • inventory_index_id (str) – Id specifying which inventory to delete.
Returns:

An expunged entry corresponding the inventory_index_id.

Return type:

InventoryIndex

Raises:

Exception – Reraises any exception.

classmethod get(session, inventory_index_id)[source]

Get an inventory index entry by id.

Parameters:
  • session (object) – Database session
  • inventory_index_id (str) – Inventory id
Returns:

Entry corresponding the id

Return type:

InventoryIndex

classmethod get_inventory_index_id_by_scanner_index_id(session, scanner_index_id)[source]

List all inventory index entries.

Parameters:
  • session (object) – Database session
  • scanner_index_id (int) – id of the scanner in scanner_index table
Returns:

inventory index id

Return type:

int64

classmethod get_inventory_indexes_older_than_cutoff(session, cutoff_datetime)[source]

Get all inventory index entries older than the cutoff.

Parameters:
  • session (object) – Database session
  • cutoff_datetime (datetime) – The cutoff point to find any older inventory index entries.
Returns:

InventoryIndex

Return type:

list

classmethod get_latest_inventory_index_id(session)[source]

List all inventory index entries.

Parameters:session (object) – Database session
Returns:inventory index id
Return type:int64
classmethod list(session)[source]

List all inventory index entries.

Parameters:session (object) – Database session
Yields:InventoryIndex – Generates each row
class Inventory(**kwargs)[source]

Bases: sqlalchemy.ext.declarative.api.Base

Resource inventory table.

__repr__()[source]

String representation of the database row object.

Returns:A description of inventory_index
Return type:str
_sa_class_manager = {'cai_resource_name': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d361a8>, 'cai_resource_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36258>, 'category': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36308>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36048>, 'inventory_errors': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36728>, 'inventory_index_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d360f8>, 'other': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36678>, 'parent_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d365c8>, 'resource_data': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36518>, 'resource_id': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d36468>, 'resource_type': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d363b8>}
cai_resource_name
cai_resource_type
category
copy_inplace(new_row)[source]

Update a database row object from a resource.

Parameters:new_row (Inventory) – the Inventory row of the new resource
classmethod from_resource(index, resource)[source]

Creates a database row object from a crawled resource.

Parameters:
Returns:

database row object.

Return type:

object

get_cai_resource_name()[source]

Get the row’s cai resource name.

Returns:cai resource name.
Return type:str
get_cai_resource_type()[source]

Get the row’s cai resource type.

Returns:cai resource type.
Return type:str
get_category()[source]

Get the row’s data category.

Returns:data category.
Return type:str
get_inventory_errors()[source]

Get the row’s error data.

Returns:row’s error data.
Return type:str
get_other()[source]

Get the row’s other data.

Returns:row’s other data.
Return type:dict
get_parent_id()[source]

Get the row’s parent id.

Returns:parent id.
Return type:int
get_resource_data()[source]

Get the row’s metadata.

Returns:row’s metadata.
Return type:dict
get_resource_data_raw()[source]

Get the row’s data json string.

Returns:row’s raw data.
Return type:str
get_resource_id()[source]

Get the row’s resource id.

Returns:resource id.
Return type:str
get_resource_type()[source]

Get the row’s resource type.

Returns:resource type.
Return type:str
static get_schema_update_actions()[source]

Maintain all the schema changes for this table.

Returns:A mapping of Action: Column.
Return type:dict
id
inventory_errors
inventory_index_id
other
parent_id
resource_data
resource_id
resource_type
class InventoryIndex(**kwargs)[source]

Bases: sqlalchemy.ext.declarative.api.Base

Represents a GCP inventory.

__repr__()[source]

Object string representation.

Returns:String representation of the object.
Return type:str
_sa_class_manager = {'completed_at_datetime': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24990>, 'counter': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24c50>, 'created_at_datetime': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d248e0>, 'id': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d247d8>, 'inventory_index_errors': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24db0>, 'inventory_index_warnings': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24d00>, 'inventory_status': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24a40>, 'message': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24e60>, 'progress': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24ba0>, 'schema_version': <sqlalchemy.orm.attributes.InstrumentedAttribute object at 0x7f73d1d24af0>}
add_warning(session, warning)[source]

Add a warning to the inventory.

Parameters:
  • session (object) – session object to work on.
  • warning (str) – Warning message
complete(status='SUCCESS')[source]

Mark the inventory as completed with a final inventory_status.

Parameters:status (str) – Final inventory_status.
completed_at_datetime
counter
classmethod create()[source]

Create a new inventory index row.

Returns:InventoryIndex row object.
Return type:object
created_at_datetime
get_details(session)[source]

Generate/return inventory details for this inventory index.

Includes delete pending/active resource types and hidden/shown datasets.

Parameters:session (object) – session object to work on.
Returns:a (resource type -> count) dictionary
Return type:dict
get_hidden_resource_details(session, resource_type)[source]

Count of the hidden and shown specified resources.

Generate/return the count of hidden resources (e.g. dataset) for this inventory index.

Parameters:
  • session (object) – session object to work on.
  • resource_type (str) – resource type to find details for.
Returns:

a (hidden_resource -> count) dictionary

Return type:

dict

get_lifecycle_state_details(session, resource_type_input)[source]

Count of lifecycle states of the specified resources.

Generate/return the count of lifecycle states (ACTIVE, DELETE_PENDING) of the specific resource type input (project, folder) for this inventory index.

Parameters:
  • session (object) – session object to work on.
  • resource_type_input (str) – resource type to get lifecycle states.
Returns:

a (lifecycle state -> count) dictionary

Return type:

dict

get_summary(session)[source]

Generate/return an inventory summary for this inventory index.

Parameters:session (object) – session object to work on.
Returns:a (resource type -> count) dictionary
Return type:dict
id
inventory_index_errors
inventory_index_warnings
inventory_status
message
progress
schema_version
set_error(session, message)[source]

Indicate a broken import.

Parameters:
  • session (object) – session object to work on.
  • message (str) – Error message to set.
class Storage(session, existing_id=0, readonly=False)[source]

Bases: google.cloud.forseti.services.inventory.base.storage.Storage

Inventory storage used during creation.

__enter__()[source]

To support with statement for auto closing.

Returns:The inventory storage object
Return type:Storage
__exit__(type_p, value, traceback)[source]

To support with statement for auto closing.

Parameters:
  • type_p (object) – Unused.
  • value (object) – Unused.
  • traceback (object) – Unused.
_create()[source]

Create a new inventory.

Returns:Index number of the created inventory.
Return type:int
Raises:Exception – Reraises any exception.
_get_resource_id(resource)[source]

Checks if a resource exists already in the inventory.

Parameters:resource (object) – Resource object to check against the db.
Returns:The resource id of the existing resource, else 0.
Return type:int
_get_resource_rows(key, resource_type)[source]

Get the rows in the database for a certain resource

Parameters:
  • key (str) – The key of the resource
  • resource_type (str) – The type of the resource
Returns:

The inventory db rows of the resource, IAM policy and GCS policy.

Return type:

object

Raises:

Exception – if there is no such row or more than one.

_open(inventory_index_id)[source]

Open an existing inventory.

Parameters:inventory_index_id (str) – the id of the inventory to open.
Returns:The inventory index db row.
Return type:object
_require_opened()[source]

Make sure the storage is in ‘open’ state.

Raises:Exception – If storage is not opened.
close()[source]

Close the storage.

Raises:Exception – If the storage was not opened before or if the storage is writeable but neither rollback nor commit has been called.
commit()[source]

Commit the stored inventory.

error(message)[source]

Store a fatal error in storage. This will help debug problems.

Parameters:message (str) – Error message describing the problem.
Raises:Exception – If the storage was opened readonly.
get_root()[source]

get the resource root from the inventory

Returns:A row in gcp_inventory of the root
Return type:object
iter(type_list=None, fetch_iam_policy=False, fetch_gcs_policy=False, fetch_dataset_policy=False, fetch_billing_info=False, fetch_enabled_apis=False, fetch_service_config=False, with_parent=False)[source]

Iterate the objects in the storage.

Parameters:
  • type_list (list) – List of types to iterate over, or [] for all.
  • fetch_iam_policy (bool) – Yield iam policies.
  • fetch_gcs_policy (bool) – Yield gcs policies.
  • fetch_dataset_policy (bool) – Yield dataset policies.
  • fetch_billing_info (bool) – Yield project billing info.
  • fetch_enabled_apis (bool) – Yield project enabled APIs info.
  • fetch_service_config (bool) – Yield container service config info.
  • with_parent (bool) – Join parent with results, yield tuples.
Yields:

object – Single row object or child/parent if ‘with_parent’ is set.

open(handle=None)[source]

Open the storage, potentially create a new index.

Parameters:handle (str) – If None, create a new index instead of opening an existing one.
Returns:Index id of the opened or created inventory.
Return type:str
Raises:Exception – if open was called more than once
rollback()[source]

Roll back the stored inventory, but keep the index entry.

type_exists(type_list=None)[source]

Check if certain types of resources exists in the inventory

Parameters:type_list (list) – List of types to check
Returns:If these types of resources exists
Return type:bool
update(resource)[source]

Update a resource in the storage.

Parameters:resource (object) – Resource object to store in db.
Raises:Exception – If storage was opened readonly.
warning(message)[source]

Store a Warning message in storage. This will help debug problems.

Parameters:message (str) – Warning message describing the problem.
Raises:Exception – If the storage was opened readonly.
write(resource)[source]

Write a resource to the storage and updates its row

Parameters:resource (object) – Resource object to store in db.
Raises:Exception – If storage was opened readonly.
initialize(engine)[source]

Create all tables in the database if not existing.

Parameters:engine (object) – Database engine to operate on.