Setup

This page explains how to set up Forseti for local development.


Before you begin

To complete this guide, you will need:

  • A Github account.
  • A Google Cloud Platform (GCP) organization.
  • A GCP project for Forseti with billing enabled.
  • The ability to assign roles on your organization’s Cloud Identity and Access Management (Cloud IAM) policy.
  • The ability to assign G Suite domain-wide delegation to the Forseti service account.

Setting up GCP infrastructure

Setting up gcloud

First, install and configure the gcloud command-line tool to run the setup commands:

  1. Download and install the gcloud command-line tool.
  2. Make sure gcloud is configured by running gcloud info and check that the Account and Project displayed match your Forseti Security project. If it doesn’t match, run the following commands to configure gcloud for your Forseti Security project:
    1. Run gcloud auth login and use your Google credentials to authenticate.
    2. Run gcloud init and select your Forseti Security project and Google account.

Creating service accounts

If you are setting up a developer environment, it’s best to use the credential from the Forseti service accounts. You can also use your own Google credentials from when you ran gcloud auth login, but your personal credentials might drift and differ from the Forseti service account.

If you are running Forseti on GCP, you’ll need to create service accounts with Cloud Identity and Access Management (Cloud IAM) roles to allow Forseti to read GCP data and to manage Forseti modules.

For a detailed explanation of how Forseti Security uses service accounts, see Forseti Service Accounts.

To create and grant roles to a service account for Forseti Inventory, Scanner, and Enforcer, follow the steps below:

  1. Go to the GCP Console and create a new service account.
  2. Create and download a JSON key for the service account.
  3. Run the following command to assume the service account credentials:
  gcloud auth activate-service-account --key-file=PATH/TO/KEYFILE.json

To enable your service account to collect G Suite data, follow the steps in Enabling G Suite Access.

Assigning roles

For Forseti to have access to read data from your GCP environment, you will need to assign the roles below to the Forseti service account or to your Google user. It is recommended that you assign roles to the service account, especially if you run Forseti in multiple environments.

Granted at the organization level

  • roles/appengine.appViewer (server)
  • roles/bigquery.metadataViewer (server)
  • roles/browser (server)
  • roles/cloudasset.viewer (server)
  • roles/cloudsql.viewer (server)
  • roles/compute.networkViewer (server)
  • roles/iam.securityReviewer (server)
  • roles/orgpolicy.policyViewer (server)
  • roles/servicemanagement.quotaViewer (server)
  • roles/serviceusage.serviceUsageConsumer (server)

Granted at the project level

  • roles/cloudsql.client (server)
  • roles/logging.logWriter (server, client)
  • roles/monitoring.metricWriter (server)
  • roles/storage.objectViewer (server, client)
  • roles/storage.objectCreator (server)

Granted at the service account level

  • roles/iam.serviceAccountTokenCreator (server)

To grant the roles on the Cloud IAM policies, use the following commands:

  • Organization: the member has access to everything under the organization. Your authorized account must have the Organization Admin role to assign the role to another member.

    To retrieve your organization id, follow these steps.

    To add Cloud IAM policy bindings to the Organization, run the following command:

    gcloud organizations add-iam-policy-binding ORGANIZATION_ID \
     --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    

    For exmple:

    gcloud organizations add-iam-policy-binding 000000000001 \
      --member=serviceAccount:service-account-01@my-foo-project.iam.gserviceaccount.com \
      --role=roles/serviceusage.serviceUsageConsumer
    
  • Folder: the member has access to everything under a particular folder. Your authorized account must have the Folder Admin role to assign the role to another member.

    To add Cloud IAM policy bindings to the Folder, run the following command:

    gcloud alpha resource-manager folders add-iam-policy-binding FOLDER_ID \
     --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    
  • Project: the member has access only to a particular project. Your authorized account must have the Owner role on the project or Folder Admin.

    To add Cloud IAM policy bindings to the Project, run the following command:

    gcloud projects add-iam-policy-binding PROJECT_ID \
     --member=MEMBER_TYPE:MEMBER_NAME --role=ROLE_NAME
    

    For example:

    gcloud projects add-iam-policy-binding my-project-name \
     --member=serviceAccount:service-account-01@my-project-name.iam.gserviceaccount.com \
     --role=roles/storage.objectCreator
    
  • Service Account: grant additional roles to the service account. Your authorized account must have the Owner role on the project that is the source of the service account.

    gcloud iam service-accounts add-iam-policy-binding YOUR_SERVICE_ACCOUNT \
      --member=serviceACcount:YOUR_SERVICE_ACCOUNT --role=ROLE_NAME
    

MEMBER_TYPE

  • Description: identity types in Cloud IAM policies
  • Valid values: user, group, serviceAccount, or domain

MEMBER_NAME

  • Description: name of the Cloud IAM member
  • Valid values: String, either a domain like example.com, or an email address like user@example.com.

Examples of MEMBER_TYPE:MEMBER_NAME:

  • user:me@gmail.com
  • serviceAccount:forseti-gcp-reader@your-project-id.iam.gserviceaccount.com
  • group:my-forseti-group@example.com
  • domain:example.com

Enabling APIs

Enable each of the required APIs by running the following command:

  gcloud services enable <API URI>
Name API URI
Admin SDK admin.googleapis.com
App Engine Admin appengine.googleapis.com
BigQuery bigquery-json.googleapis.com
Cloud Asset cloudasset.googleapis.com
Cloud Billing cloudbilling.googleapis.com
Cloud IAM iam.googleapis.com
Cloud SQL Administration sqladmin.googleapis.com
Cloud SQL sql-component.googleapis.com
Cloud Storage storage-api.googleapis.com
Compute Engine compute.googleapis.com
Deployment Manager deploymentmanager.googleapis.com
Groups Settings groupssettings.googleapis.com
Kubernetes Engine container.googleapis.com
Resource Manager cloudresourcemanager.googleapis.com
Stackdriver Logging logging.googleapis.com
Service Management servicemanagement.googleapis.com
Service Usage serviceusage.googleapis.com

Setting up Cloud SQL

Forseti stores data in Cloud SQL. You can connect to the Cloud SQL instance by using the Cloud SQL proxy to authenticate to GCP with your Google credentials, instead of opening up network access to your Cloud SQL instance. To set up your Cloud SQL instance for Forseti, follow the steps below:

  1. Go to the Google Cloud Platform Console SQL page and follow the steps below to create a new instance:
    1. Select a MySQL database engine.
    2. Select a Second Generation instance type.
    3. If you see a Choose use case page, select the configuration type you want based upon whether this is for production or development use.
    4. On the Create a MySQL Second Generation instance page, enter an Instance ID and Root password, then select the following settings:
      1. Database version: MySQL 5.7
      2. Machine type: db-n1-standard-1 machine type
      3. Storage capacity: 25 GB
    5. Add or modify other database details as you wish.
    6. When you’re finished setting up the database, click Create.
  2. Create a new user, like forseti_user, with read/write privileges for Forseti to access the database. Don’t set a password for the new user. This will allow Cloud SQL Proxy to handle authentication to your instance.
  3. Create a new database, like forseti_security.
  4. Use the SQL Proxy to proxy your connection to your Cloud SQL instance. Your INSTANCE_CONNECTION_NAME is the Instance Connection Name under Properties on the Cloud SQL dashboard instance details, in the format “PROJECTID:REGION:INSTANCEID”.

    If you are using Cloud SDK authentication:

       <path/to/cloud_sql_proxy> -instances=INSTANCE_CONNECTION_NAME=tcp:3306
    

    If you are using a service account to authenticate (recommended for production environments):

       <path/to/cloud_sql_proxy> \
           -instances=<INSTANCE_CONNECTION_NAME>=tcp:3306 \
           -credential_file=<PATH_TO_KEY_FILE>
    

    For example:

       ./cloud_sql_proxy \
           -instances=foo-project-name:us-central1:mysql-instance=tcp:3306 \
           -credential_file=/usr/local/google/home/foo/foo-project-cert-0d152c0e1bc8.json
    
  5. If you are setting up a development environment, install MySQL Workbench. This is a GUI tool that will make it much easier to view and query the Forseti tables and data.

    1. Connection Name
    2. Hostname: 127.0.0.1
    3. Port: 3306
    4. Username: forseti_user

Setting up a local environment

Ubuntu setup

Install the necessary python dev tools and packages from apt_packages.txt.

Mac setup

This guide is written for use with Homebrew.

Use the following commands to install the necessary dependencies:

Install python-dev:

The system python that comes with OS X does not allow certain packages (such as six) to be modified. Install brew’s version of python so that separate python packages can be installed and managed.

  brew install python@3

Install openssl:

  brew install openssl

Install mysql_config:

  brew install mysql

Creating a virtualenv

Ensure virtualenv is installed in your system. Virtualenv allows you to create multiple environments to contain different modules and dependencies in different projects:

  sudo pip3 install virtualenv

Use the following command to create a virtualenv:

  # create a virtualenv
  mkvirtualenv forseti-security

  workon forseti-security

Getting the source code

Follow our contributing guidelines to create a fork of the Forseti code, and learn how to submit a pull request (PR).

Installing build dependencies

Use the following command to navigate to your cloned repository and install required build dependencies:

  cd forseti-security

  pip3 install -q --upgrade -r requirements.txt

Running the Python setup

Use the following commands to run the Python setup:

  python3 setup.py install

Configuring Forseti settings

Before you run Forseti, you need to edit the forseti_conf_server.yaml file. For more information, see Configuring Forseti.

Starting Forseti

After you complete the above steps, you should be able to run the Forseti server and the command-line interface (CLI) client:

  forseti_server \
  --endpoint "localhost:50051" \
  --forseti_db "mysql+pymysql://root@127.0.0.1:3306/forseti_security" \
  --services scanner model inventory explain notifier \
  --config_file_path "PATH_TO_YOUR_CONFIG.yaml" \
  --log_level=info \
  --enable_console_log

To test the server is healthy and running, run the following command in another terminal window , and see if the server responds:

  forseti inventory list

Troubleshooting

If you are installing on Mac OS X with Homebrew and get a fatal error related to 'openssl/opensslv.h' file not found, you might need to export CPPFLAGS and LDFLAGS for the openssl package. For more information, see issue 3489.

To find the CPPFLAGS and LDFLAGS information and export them, run the following command:

  brew info openssl

  ... lots of information ...

  There aren't usually any consequences of this for you. If you build your
  own software and it requires this formula, you'll need to add to your
  build variables:

  LDFLAGS:  -L/SOME/PATH/TO/openssl/lib
  CPPFLAGS: -I/SOME/PATH/TO/openssl/include

Next, copy the LDFLAGS and CPPFLAGS values and export them, similar to the following:

  export CPPFLAGS=-I/SOME/PATH/TO/openssl/include

  export LDFLAGS=-L/SOME/PATH/TO/openssl/lib

In the above example, /SOME/PATH/TO represents the path specific to your system. Make sure to use the values from your terminal.


If on Linux executing mkvirtualenv forseti-security results in bash: mkvirtualenv: command not found then try the following:

sudo pip3 install virtualenv virtualenvwrapper

Now attempt to make a virtual environment again.


If on Linux executing workon forseti-security results in bash: workon: command not found then ensure workon is in the source path. Try fixing source path issue by executing the following:

source /usr/local/bin/virtualenvwrapper.sh

and then trying workon forseti-security again.

You can also put these into your bash profile:

  export WORKON_HOME=$HOME/.virtualenvs
  source /usr/local/bin/virtualenvwrapper.sh

If on Linux executing python3 setup.py install results in EnvironmentError: mysql_config not found then try the following:

sudo apt install default-libmysqlclient-dev


If on OS X executing python3 setup.py install results in my_config.h not found, then try the following:

source

  brew install mysql

  brew unlink mysql

  brew install mysql-connector-c

  sed -i -e 's/libs="$libs -l "/libs="$libs -lmysqlclient -lssl -lcrypto"/g' /usr/local/bin/mysql_config

  pip3 install MySQL-python

  brew unlink mysql-connector-c

  brew link --overwrite mysql
  

If running the server results in this error:

ImportError: No module named forseti.common.util

Try to add a symlink from the source code to the virtual environment:

cd <virtual environment>/lib/python3.6/site-packages/google/cloud

ln -s <path to your git source code>/google/cloud/forseti forseti

What’s next

  • To learn about more CLI commands, see Use.