Proposal for implementation of security (encryption and RBAC) for Pravega

Status: Under Discussion

Related issues:

Motivation

Pravega is built for enterprise use from day one. Enterprises really care of security and the access control of the data. This document proposes an approach to implementing security for Pravega.

Aspects of security

1. Encryption

a. Encryption of data at rest (Tier 2)

All the Tier 2 used have mechanisms for which encryption and access control can be enabled. For a secure deployment, Pravega should use these mechanisms. The configuration and credentials for doing this should be passed on to Pravega deployment in a secure manner.

b. Encryption of data in flight (over network and in Tier 1)

https://github.com/arvindkandhare/pravega/blob/pdp25-images/images/PDP25-PravegaEncryption.jpg

1. All the nodes should authenticate to each other using X.509 certificates. A sample self singed root cert will be provided along with Pravega package. Pravega also will share scripts using which server authentication certificates can be created for Pravega nodes using this root certificate. Deployers are expected to create their own cert instead of this sample certificate in a production deployment.

2. Traffic between the nodes should be encrypted using SSL/TLS

c. Encryption of data at rest in Tier 1

1. Tier-1 based on Apache Bookkeeper stores data on disks. This is a temporary storage and it is removed from bookkeeper storage once the data is moved to Tier 2.
2. Pravega segmentstore uses rocksdb for local cache on each node. This cache is configured to spill over to disk. This data is also not encrypted. Pravega segmentstore cleans up this directory on clean exit.

Adding encryption to both these components will be costly as they are on the high performance path. It is assumed that this data is secure as the time data spends on disk in these scenarios is very less. This assumption might not be correct. This is still an open issue. This github issue tracks the future work in this direction: https://github.com/pravega/pravega/issues/2224

d. Communication between different components need to be secured using the existing mechanisms provided with the components

1. Communication between Pravega SegmentStore/Controller and ZK Here are more details about how ZK can be deployed securely. https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+SSL+User+Guide and this page describes how ZK can be deployed with authorization enabled: https://cwiki.apache.org/confluence/display/ZOOKEEPER/ZooKeeper+and+SASL The parameters to connect to ZK is sent through configuration class ServiceConfig in case of Pravega segmentstore. It is sent through ZkClientConfig in case of Pravega Controller. These classes should take necessary parameters for secure deployment.

2. Communication between Pravega controller and Pravega segmentstore Please refer to the next section for more details on how encryption, authorization and authentication is enabled for the communication between Pravega Controller and Pravega Segmentstore.

3. Communication between Pravega SegmentStore and BK Here is a link to how BK is deployed securely. https://bookkeeper.apache.org/docs/4.5.0/security/overview/ BookKeeper can be deployed either in a secure OR insecure manner. The deployment can also support mix of authenticated, unauthenticated, encrypted and non-encrypted approaches. Pravega should support BookKeeper deployment in each of these combinations. From Pravega side it means configuring Pravega client with all the possible parameters supporting this. These parameters are passed to Pravega segmentstore deployment through the configuration class BookKeeperConfig. This class should support new configurations to support secure communication between segment store and BK.

4. Communication between BK and ZK BookKeeper supports secure communications through ZK. Pravega does not proscribe a single way of deploying BK and BK. The deployments should ensure that these components are deployed securely and the communication between them is secure.

2. Role based access control

1. Credentials for Tier-1 and Tier-2

Options:

1. Carry the credentials of the user running the operation to Tier-1 and Tier-2. The stream segments should be owned by the user who has created the stream
2. Pravega uses service credentials to create and manage data in Tier-1 and tier-2.

The plan is to use option 2.

Rationale:

1. The credential management of Pravega users and of tier-2 implementations can be completely different. In some combinations, it might not be possible at all to convert the user from one approach to another.
2. Tier-2 operations can happen completely out of sync with user interactions with Pravega. User context may not be available during the delayed operations.

3. Authorization:

Pravega will implement a pluggable authorization model. The authorization kicks in only with interactions with the controller through GRPC or REST. Once the request is authorized, controller generates a token. This token will be presented to the SegmentStore. Once the token is validated, SegmentStore will assume that the interactions have already been approved by the controller. The authorizer model returns whether the user is authorized as well as a string which represents the user identity. https://github.com/arvindkandhare/pravega/blob/pdp25-images/images/PDP25-PravegaAuth.jpg Advantages: This will mean that we do not have authorization happening twice, once with the controller and then with segmentstore. This will also take the authorization part away from the performance critical path.

a. Token format

Token is used to share authorization information between Pravega controller and SegmentStore. The token follows the JWT (JSON Web Token) format closely. https://tools.ietf.org/html/rfc7519 It is signed by a symmetric key shared between controller and SegmentStore. More details about how JWT tokens are signed can be found here: https://tools.ietf.org/html/rfc7515. A token consists of the resource identifier. SegmentStore has the responsibility of validating the token. It also converts the token from controller primitives (stream/scope) to SegmentStore primitives (segments). SegmentStore validates signature of this token, validates that the resource id requested matches the one specified in the token, validates the lifetime of the token and if it matches, performs the given operation.

b. Why JWT

Jason Web Token format and implementation gives an efficient way of signing and encrypting claim based tokens. This is a widely used format and has advanced features in-built like token expiry etc which will be useful in the context of Pravega. Please refer to JWT specs for more details.

c. Token lifetime and revocation

Tokens are short lived. Controller controls their lifetime using the 'exp' claim. https://tools.ietf.org/html/rfc7519#section-4.1.4. The expiration time is checked before start of a given SegmentStore operation. In case an expired token is observed, SegmentStore returns appropriate error to the client. The client can interact with controller and receive a new token after controller reauthenticates it.

4. User access to Pravega resources (scopes, streams and reader groups)

The default implementation of the authorization module, Controller manages mapping of resources to ids. The underlying authorizer implementation just returns a validated user identifier. Controller stores a table containing resource name, ACLs and user id securely. Controller will have APIs to add this table is also one of the resources. Using this entries can be made into the table. A stream inherits the RBAC behavior of the scope it belongs to. If the behavior is exclusively specified for the stream, it has precedence over the inherited access control definition. The default implementation does not have the concept of group and group membership.

5. Interface changes

1. Motivation and approach behind interface changes

Pravega deployments and client implementations should be able to switch between secure and insecure version with very little/no changes. Pravega deployment should be possible with different combinations of encryption, authentication and authorization. Pravega should also support drop in authorization module where custom authorization can be implemented. Multiple such authorization/authentication schemes may co-exist in a single Pravega deployment. These changes are done with consideration to following points:

a. Ease of use b. Ease of automation c. Same approach for insecure and secure API d. Logical switch between insecure and secure version of API

2. Configuration changes

Pravega controller has new configurations which tell it to use encryption and authorization. Each authorization implementation is represented by a unique name. Default Pravega implementation will be known as "Pravega-Default". Path to x.509 certificate and its private key will also be part of the newly added configuration.

3. Pluggable authorization/authentication

Apart from "Pravega-Default", other custom implementations of the authorizer interface can exist. Pravega will pick up jars containing other custom implementations of the authorization class from a path mentioned in the configuration.

4. Client API changes

Parameters related to encryption and authorization/authentication that are sent to the client are encapsulated in a PravegaClientConfig object.

public class PravegaClientConfig {
    /** controllerURI The controller rpc URI. This can be of 2 types
     1. tcp://ip1:port1,ip2:port2,...
        This is used if the controller endpoints are static and can be directly accessed.
     2. pravega://ip1:port1,ip2:port2,...
        This is used to autodiscovery the controller endpoints from an initial controller list.
    */
    private final URI controllerURI;
    /**
     * Flag to enable TLS. TODO: This can be read from the URL.
     */
    private final boolean enableTls;
    /**
     * Credentials to be passed on to the Pravega controller for authentication and authorization.
     */
    private final PravegaCredentials credentials;
    /**
     * Path to an optional truststore. If this is null or empty, the default JVM trust store is used.
     */
    private final String pravegaTrustStore;
}

Client API are updated to accept this object as parameter. Here is an updated list of client APIs:

1. ClientFactory::withScope

 /**
     * Creates a new instance of Client Factory.
     *
     * @param scope The scope string.
     * @param config Configuration for the client.
     * @return Instance of ClientFactory implementation.
     */
    static ClientFactory withScope(String scope, PravegaClientConfig config) 

2. StreamManager::create

/**
     * Creates a new instance of StreamManager.
     *
     * @param clientConfig Configuration for the client connection to Pravega.
     * @return Instance of Stream Manager implementation.
     */
    public static StreamManager create(PravegaClientConfig clientConfig) 

3. ReaderGroupManager::withScope

/**
     * Creates a new instance of ReaderGroupManager.
     *
     * @param scope The Scope string.
     * @param clientConfig Configuration for the client.
     * @return Instance of Stream Manager implementation.
     */
    public static ReaderGroupManager withScope(String scope, PravegaClientConfig clientConfig) 

4.1 Rationale for client API changes

Pravega client uses configuration objects for configuration of different aspects of Pravega client. e.g. ReaderConfig, SynchronizerConfig. With security related changes, we introduce one more configuration object: PravegaClientConfig. This represents the parameters which are constant throughout the life time of the ClientFactory object. For a single client factory, the controller URL and the approach to security stays for the life time of the object. Because of this, these parameters are passed through static creator methods for ClientFactory: withScope. The option of adding these to reader/writer config objects is rejected.

6. Implementation changes

1. Controller

a. GRPC interface changes

Most of the GRPC calls return a delegation token along with the regular return structure. Each GRPC handler function takes the following steps:

1. It calls the authenticateExecuteAndProcessResults function.
2. This function hands over the custom headers to appropriate auth handler through PravegaAuthHandler
3. In case of successful authorization the actual function is called.
4. A valid delegation token is created and handed over to the client along with the actual result of the API.
5. Any failure in the authentication/authorization process leads to connection drop.

b. REST interface changes

1. In case of REST interface, it is expected that the same custom headers are sent through the http headers.

c. Task engine changes

1. SegmentStore task engine creates a single delegation token with wild-card permissions.
2. This token is used for all the interactions with the SegmentStore
3. Any auth error is treated as a retryable exception.

2. SegmentStore

a. PravegaRequestProcessor changes:

PravegaRequestProcessor validates each incoming request through a tokenVerifier. In case of failure of token verification segmentstore returns a AuthTokenCheckFailed response and disconnects the connection.

3. Client implementation changes

a. Handling of delegation token

Client retrieves delegation token from the recent interactions with the controller and uses it to interact with segmentstore.

b. Auth related exception handling

Any Authentication failure from segmentstore is treated as a connection dropped. In most of the cases when connection dropped error client implementation retrieves the data from the controller before setting up a new connection with segmentstore. This interaction will refresh the token. In this way the client will have the latest token.

7. Implementation of custom auth implementation

Pravega allows for custom implementation of auth API to be picked up dynamically. These jars are deployed with controller. Rest of the flow remains the same.

Here are the steps involved in implementing a custom auth handler:

1. Implement the following interface from Pravega client in a custom jar. The implementations are loaded from the classpath using ServiceLoader (https://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html) Pravega controller also has one default implementation of implements this interface: PravegaAuthHandler.

public interface PravegaAuthHandler {

    enum  PravegaAccessControlEnum {
        NONE,
        READ,
        READ_UPDATE
    }

    /**
     * Returns name of the handler. Only the first implementation with a unique name will be loaded.
     * @return The unique name assigned to the handler.
     */
    String getHandlerName();

    /**
     * Authenticates a given request. Pravega controller passes the HTTP headers associated with the call.
     * The custom implementation returns whether the user represented by these headers is authenticated.
     *
     * @param headers the key-value pairs passed through grpc.
     * @return Returns true when the user is authenticated.
     */
    boolean authenticate(Map<String, String> headers);

    /**
     * Authorizes the access to a given resources. Pravega controller passes the HTTP headers associated with the call.
     * The implementations of this interface should return the maximum level of authorization possible for the user represented
     * by the headers.
     *
     * @param resource the resource that needs to be accessed.
     * @param headers the context for authorization.
     * @return The level of authorization. Throws exception if not authorized.
     */
    PravegaAccessControlEnum authorize(String resource, Map<String, String> headers);

    /**
     * Sets the configuration. If the auth handler needs to access the server configuration, it can be accessed though this var.
     *
     * @param serverConfig The server configuration.
     */
    void setServerConfig(Object serverConfig);

2. These binaries are placed in CLASSPATH for Pravega controller.

3. Create an implementation of PravegaCredentials interface: A client selects its auth handler by setting a grpc header with a name "method" the value should match the name of the custom implementation. This is done by implementing PravegaCredentials interface and passing it to client calls.

8. Administrator guide to deploy Pravega in a secure manner

1. Deploying Dependent components in a secure manner

All the external components should be deployed securely. These components will include Apache Bookkeeper, Apache Zookeeper, HDFS (or other options for Tier-2). Refer to individual components deployment guide for this.

2. Creating server auth certificates:

Server auth certificates are created for nodes on which Pravega Segmentstore and controller are installed Use certificate generation scripts delivered with Pravega to achieve this.

3. Configure Pravega Segmentstore and controller with security settings

Refer to configuration guide for the settings that needs to be applied.

4. Install the public certificate used to sign the server certs on client nodes.

This can be either installed to a custom trust store OR the default trust store. In case a custom trust store is used, it should be mentioned in the PravegaClientConfig object passed on to the client.

9. Threat Modelling

1. Application decomposition

a. External dependencies

b. Entry points

c. Assets

d. Trust levels

e. Data flow diagrams

2. List of threats and ranking of threats

a. List of threats

b. Categorization of threats

c. Security controls : Authentication, Authorization, token management, data input validation, error handling, logging, information auditing, cryptography, session management

3. Countermeasures and mitigations

10. further reading

1. Authorization with oauth2 and grpc

https://grpc.io/docs/guides/auth.html#authenticate-with-google-using-an-oauth2-token

2. Application threat modelling

https://www.owasp.org/index.php/Application_Threat_Modeling