Development Environment Setup

This page explains how to set up Forseti for local development.

Before you begin

To complete this guide, you will need:

  • A GCP organization.
  • A GCP project (in above organization) for Forseti Security with billing enabled.
  • The ability to assign roles on your organization’s Cloud IAM policy.

Setting GCP infrastructure

Setting up gcloud

First, install and configure the gcloud command-line tool so you can run the setup commands:

  1. Download and install the gcloud command-line tool.
  2. Make sure gcloud is configured by running gcloud info and check that the Account and Project displayed match your Forseti Security project. If it doesn’t match, run the following commands to configure gcloud for your Forseti Security project:
    1. Run gcloud auth login and use your Google credentials to authenticate.
    2. Run gcloud init and select your Forseti Security project and Google account.

Creating service accounts

If you are setting up a developer environment, you can just use the Google credentials from when you ran gcloud auth login and go to the next section.

If you are running Forseti on GCP, you’ll need create service accounts with Cloud Identity and Access Management (Cloud IAM) roles to allow Forseti to read GCP data and to manage Forseti modules.

For a detailed explanation of how Forseti Security uses service accounts, refer to “Forseti Service Accounts”.

To create and grant roles to a service account for Forseti Inventory, Scanner, and Enforcer, follow the steps below.

  1. Go to your Google Cloud Platform console and create a new service account.
  2. Create and download a json key for the service account.
  3. Run the following command to assume the service account credentials:
  gcloud auth activate-service-account --key-file=PATH/TO/KEYFILE.json

To create a separate service account for enabling G Suite data collection, follow the steps in “Enabling GSuite Google Groups Collection”.

Assigning roles

In order for Forseti to have access to read data from your GCP environment, you will need to assign roles to a particular member: either the Inventory/Scanner/Enforcer service account or your Google user. You can refer to the official documentation about members for more information.

Also, you can grant the roles on the organization, folder, or project IAM policies.

  • Organization IAM: the member has access to the everything under the organization. Your authed account must have the Organization Admin role to assign the role to another member.

    The command to add IAM policy bindings to the organization IAM is:

    $ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
       --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    
  • Folder IAM: the member has access to everything under a particular folder. Your authed account must have the Folder IAM Admin role to assign the role to another member.

    The command to add IAM policy bindings to the folder IAM is:

    $ gcloud alpha resource-manager folders add-iam-policy-binding FOLDER_ID \
       --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    
  • Project IAM: the member has access only to a particular project. Your authed account must either have the Owner role on the project or Folder IAM Admin.

    The command to add IAM policy bindings to the folder IAM is:

    $ gcloud projects add-iam-policy-binding PROJECT_ID \
       --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    

The MEMBER_TYPE value is either user, group, serviceAccount, or domain.

The MEMBER_NAME is either a domain (e.g. example.com) or an email address (user@example.com).

Examples of MEMBER_TYPE:MEMBER_NAME:

  • user:me@gmail.com
  • serviceAccount:forseti-gcp-reader@your-project-id.iam.gserviceaccount.com
  • group:my-forseti-group@example.com
  • domain:example.com

Use these commands to grant the Forseti roles to your organization IAM policy. If you need to assign the roles on the folder or project level, use the commands from above, with the roles below.

$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/browser
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/compute.networkViewer
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/iam.securityReviewer
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/appengine.appViewer
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/servicemanagement.quotaViewer
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/cloudsql.viewer
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/compute.securityAdmin
$ gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/bigquery.dataViewer

Project Cloud IAM roles

These are necessary for reading/writing Forseti data in Google Cloud Storage and Cloud SQL. Do not assign these outside of the project IAM.

$ gcloud projects add-iam-policy-binding FORSETI_PROJECT_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/storage.objectViewer
$ gcloud rpojects add-iam-policy-binding FORSETI_PROJECT_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/storage.objectCreator
$ gcloud projects add-iam-policy-binding FORSETI_PROJECT_ID \
--member=MEMBER_TYPE:MEMBER_NAME \
--role=roles/cloudsql.client

Enabling APIs

Enable each of the required APIs by running the following command:

  $ gcloud beta service-management enable <API URI>
Name API URI
Admin SDK API admin.googleapis.com
AppEngine Admin API appengine.googleapis.com
Cloud Resource Manager API cloudresourcemanager.googleapis.com
Cloud SQL Admin API sqladmin.googleapis.com
Cloud SQL API sql-component.googleapis.com
Compute Engine API compute.googleapis.com
Deployment Manager API deploymentmanager.googleapis.com

Setting up Cloud SQL

Forseti stores data in Cloud SQL. You can connect to the Cloud SQL instance by using the Cloud SQL proxy to authenticate to GCP with your Google credentials, instead of opening up network access to your Cloud SQL instance. To set up your Cloud SQL instance for Forseti, follow the steps below:

  1. Go to the Cloud Console SQL page and follow the steps below to create a new instance:
    1. Select a MySQL database engine.
    2. Select a Second Generation instance type.
    3. On the Create a MySQL Second Generation instance page, enter an Instance ID and Root password, then select the following settings:
      1. Database version: MySQL 5.7
      2. Machine type: db-n1-standard-1 machine type
      3. Storage capacity: 25 GB
    4. Add or modify other database details as you wish.
    5. When you’re finished setting up the database, click Create.
  2. Create a new user, e.g. forseti_user, with read/write privileges for Forseti to access the database. Don’t set a password for the new user. This will allow Cloud SQL Proxy to handle authentication to your instance.
  3. Create a new database, e.g. forseti_security.
  4. Use the SQL Proxy to proxy your connection to your Cloud SQL instance. Your INSTANCE_CONNECTION_NAME is the Instance Connection Name under Properties on the Cloud SQL dashboard instance details, with the format “PROJECTID:REGION:INSTANCEID”.

      $ <path/to/cloud_sql_proxy> -instances=INSTANCE_CONNECTION_NAME=tcp:3306
    
  5. Make a note of your the Cloud SQL user you created (e.g. “forseti_user”) as well as the database name (e.g. “forseti_security” – this is NOT the ID of your Cloud SQL instance). You will need these for your forseti_conf.yaml later.

Setting up local environment

Ubuntu setup

Install the necessary python dev tools using the following command:

$ sudo apt-get install python-pip python-dev

Mac setup

This guide makes a very light assumption that you have Homebrew.

Use the following command to install python (which will also install the necessary python dev tools):

$ brew install python

Also install openssl:

$ brew install openssl

Installing mysql_config

The MySql-python library requires the mysql_config utility to be present in your system. Following are example commands to install mysql_config:

  # Ubuntu
  # Note: If libmysqlclient-dev doesn't install `mysql_config`, then try also installing `mysql_server`.
  $ sudo apt-get install libmysqlclient-dev

  # OSX, using homebrew
  $ brew install mysql

Creating a virtualenv

Use the following command to create a virtualenv:

  # create a virtualenv
  $ mkvirtualenv forseti-security
  $ workon forseti-security

Getting the source code

Use the command below to get the Forseti code if you haven’t already:

  $ git clone https://github.com/GoogleCloudPlatform/forseti-security.git

Installing build dependencies

Use the following command to install required build dependencies:

  $ pip install --upgrade \
    coverage \
    codecov \
    google-apputils \
    grpcio==1.4.0 \
    grpcio-tools==1.4.0 \
    mock \
    netaddr \
    parameterized \
    pylint

Running the python setup

Use the following commands to navigate to your cloned repo and run the python setup:

  $ cd forseti-security
  $ python setup.py install

Troubleshooting

If you are installing on Mac OS X with Homebrew and see a fatal error related to 'openssl/opensslv.h' file not found, you may need to export CPPFLAGS and LDFLAGS for the openssl package (see this issue for more information). You can find the CPPFLAGS and LDFLAGS information and export them as follows:

  $ brew info openssl
  
    ... lots of information ...
    
    Generally there are no consequences of this for you. If you build your
    own software and it requires this formula, you'll need to add to your
    build variables:

    LDFLAGS:  -L/SOME/PATH/TO/openssl/lib
    CPPFLAGS: -I/SOME/PATH/TO/openssl/include

Then copy the LDFLAGS and CPPFLAGS values and export them, similar to the following (use the values from your terminal, not “/SOME/PATH/TO”):

  $ export CPPFLAGS=-I/SOME/PATH/TO/openssl/include
  $ export LDFLAGS=-L/SOME/PATH/TO/openssl/lib

Configuring Forseti settings

Before you run Forseti, you need to edit the forseti_conf.yaml file, found in forseti-security/configs/forseti_conf.yaml. Refer to “Configuring Forseti” for more information.

Executing Forseti commands

After you complete the above steps, you should be able to run the following command-line tools:

  • forseti_inventory
  • forseti_scanner
  • forseti_enforcer
  • forseti_notifier
  • forseti_api
  • forseti_iam

To display the flag options for each tool, use the --helpshort or --helpfull flags.