This page describes how to update Forseti Inventory to collect and store new data types, and import it into a data model.
To add a new type of Inventory data, you’ll complete the following tasks:
The following guide demonstrates these steps as a walkthrough of PR #883, which adds Compute Engine Image data to Inventory and a data model.
To check if the API client to retrieve the data already exists, look at the
SUPPORTED_APIS
map in _supported_apis.py.
If the API client isn’t there, you will have to add it. For a self-contained example,
see cloud_sql.py.
Edit
google/cloud/forseti/services/inventory/base/gcp.py
to create iter_foo()
that will call the API client to retrieve the data.
@create_lazy('compute', _create_compute)
def iter_images(self, projectid):
"""Image Iterator from Compte Engine API call
Yields:
dict: Generator of image resources
"""
for image in self.compute.get_images(projectid):
yield image
Edit
google/cloud/forseti/services/inventory/base/resources.py
to create FooIterator
to call the iter_foo()
, and cast the result for storage
in Inventory.
class ImageIterator(ResourceIterator):
def iter(self):
gcp = self.client
if (self.resource.enumerable() and
self.resource.compute_api_enabled(gcp)):
for data in gcp.iter_images(
projectid=self.resource['projectId']):
yield FACTORIES['image'].create_new(data)
To complete the casting, edit
google/cloud/forseti/services/inventory/base/resources.py
to create a resource class for foo. This allows you to access the id
and type
.
If the resource doesn’t have a provided id
, you’ll have to synthesize one. For
details to create a synthetic key, see an existing key()
in resources.py
where other existing resource attributes are hashed.
class Image(Resource):
def key(self):
return self['id']
def type(self):
return 'image'
Edit
google/cloud/forseti/services/inventory/base/resources.py
to create ResourceFactory
for foo
, and link the FooIterator
to the parent resource.
FACTORIES = {
'project': ResourceFactory({
'dependsOn': ['organization', 'folder'],
'cls': Project,
'contains': [
ImageIterator,
FirewallIterator,
NetworkIterator,
SubnetworkIterator,
]}),
'image': ResourceFactory({
'dependsOn': ['project'],
'cls': Image,
'contains': [
]}),
}
This step assumes that you’re working with a simple resource that will go into the resource data model table. If you want to convert the Inventory data into a more complicated data model, email discuss@forsetisecurity.org for help.
foo
into gcp_type_list
.
def run(self):
"""Runs the import.
Raises:
NotImplementedError: If the importer encounters an unknown
inventory type.
"""
gcp_type_list = [
'organization',
'folder',
'project',
'image',
]
_convert_foo()
to store the Inventory data in a data model.
def _convert_image(self, image):
"""Convert a image to a database object.
Args:
image (object): Image to store.
"""
data = image.get_data()
parent, full_res_name, type_name = self._full_resource_name(
image)
self.session.add(
self.dao.TBL_RESOURCE(
full_name=full_res_name,
type_name=type_name,
name=image.get_key(),
type=image.get_type(),
display_name=data.get('displayName', ''),
email=data.get('email', ''),
data=image.get_data_raw(),
parent=parent))
foo
with the _convert_foo()
in the handlers
map in
_store_resource()
.
handlers = {
'organization': (None,
self._convert_organization,
None),
'folder': (None,
self._convert_folder,
None),
'project': (None,
self._convert_project,
None),
'image': (None,
self._convert_image,
None),
}
Your new data type is now added to Inventory and a new data model.
To exercise unit tests for Inventory resources, there are some existing unit tests that use mock data to verify that the Inventory modules are working correctly together. These tests will verify that the number of resources match the expected value.
__init__
methodstart
methodstop
method_mock_cloudsql
method provides a short and straight forward example.test_crawling_to_memory_storage
test to ensure that the mock
resources are accounted for in the inventory.
GCP_API_RESOURCES
variable.Now that the new Inventory resource will be included in the test data, the
test_crawling_to_memory_storage
test can be run by following the
Testing instructions.