This page describes how to update Forseti Inventory to collect and store new data types, and import it into a data model.
To add a new type of Inventory data, you’ll complete the following tasks:
The following guide demonstrates these steps as a walkthrough of PR #883, which adds Compute Engine Image data to Inventory and a data model.
To check if the API client to retrieve the data already exists, look at the
SUPPORTED_APIS
map in _supported_apis.py.
If the API client isn’t there, you will have to add it. For a self-contained example,
see cloud_sql.py.
Edit
google/cloud/forseti/services/inventory/base/gcp.py
to create iter_foo()
that will call the API client to retrieve the data.
@create_lazy('compute', _create_compute)
def iter_images(self, projectid):
"""Image Iterator from Compte Engine API call
Yields:
dict: Generator of image resources
"""
for image in self.compute.get_images(projectid):
yield image
Edit
google/cloud/forseti/services/inventory/base/resources.py
to create FooIterator
to call the iter_foo()
, and cast the result for storage
in Inventory.
class ImageIterator(ResourceIterator):
def iter(self):
gcp = self.client
if (self.resource.enumerable() and
self.resource.compute_api_enabled(gcp)):
for data in gcp.iter_images(
projectid=self.resource['projectId']):
yield FACTORIES['image'].create_new(data)
To complete the casting, edit
google/cloud/forseti/services/inventory/base/resources.py
to create a resource class for foo. This allows you to access the id
and type
.
If the resource doesn’t have a provided id
, you’ll have to synthesize one. For
details to create a synthetic key, see an existing key()
in resources.py
where other existing resource attributes are hashed.
class Image(Resource):
def key(self):
return self['id']
def type(self):
return 'image'
Edit
google/cloud/forseti/services/inventory/base/resources.py
to create ResourceFactory
for foo
, and link the FooIterator
to the parent resource.
FACTORIES = {
'project': ResourceFactory({
'dependsOn': ['organization', 'folder'],
'cls': Project,
'contains': [
ImageIterator,
FirewallIterator,
NetworkIterator,
SubnetworkIterator,
]}),
'image': ResourceFactory({
'dependsOn': ['project'],
'cls': Image,
'contains': [
]}),
}
This step assumes that you’re working with a simple resource that will go into the resource data model table. If you want to convert the Inventory data into a more complicated data model, email discuss@forsetisecurity.org for help.
foo
into gcp_type_list
.
def run(self):
"""Runs the import.
Raises:
NotImplementedError: If the importer encounters an unknown
inventory type.
"""
gcp_type_list = [
'organization',
'folder',
'project',
'image',
]
_convert_foo()
to store the Inventory data in a data model.
def _convert_image(self, image):
"""Convert a image to a database object.
Args:
image (object): Image to store.
"""
data = image.get_data()
parent, full_res_name, type_name = self._full_resource_name(
image)
self.session.add(
self.dao.TBL_RESOURCE(
full_name=full_res_name,
type_name=type_name,
name=image.get_key(),
type=image.get_type(),
display_name=data.get('displayName', ''),
email=data.get('email', ''),
data=image.get_data_raw(),
parent=parent))
foo
with the _convert_foo()
in the handlers
map in
_store_resource()
.
handlers = {
'organization': (None,
self._convert_organization,
None),
'folder': (None,
self._convert_folder,
None),
'project': (None,
self._convert_project,
None),
'image': (None,
self._convert_image,
None),
}
Your new data type is now added to Inventory and a new data model.