Example integration of Feast feature store with Valohai MLOps platform.
This example assumes PostgresSQL backend for Feast, but can be modified to work with any backend(s).
ML models don't like raw data, but refined data. Data scientists call the refined data features.
A feature store defines:
- What features are available
- Where is the the raw data stored
- How features are transformed from the raw data
- Where the refined features data will be stored
Feature store offers ability to:
- Pull training data (slow, high latency)
- Pull inference data (fast, low latency)
While feature stores are commonly services, Feast is really not.
Feast is "just" a Python library (SDK).
With Feast SDK, you define features in Python and update them into your data store.
You can also query the features back as a batch (offline / training) or single entity (online / deployment).
Valohai steps available:
- initialize - Uploads the example raw data into the SQL server
- apply - Update changed feature definitions into the store. Usually triggered by new commit in the code repository.
- materialize - Update precalculated data into the online store. Usually scheduled daily by CI/CD.
- fetch (offline) - Fetch a batch of historical features for training a model
- fetch (online) - Fetch features of single entity for model inference
- Make sure you have a Postgres SQL server which is accessible from an Valohai execution.
- Set these environment variables in your Valohai project:
FEAST_POSTGRES_HOSTNAMEFEAST_POSTGRES_USERNAMEFEAST_POSTGRES_PASSWORDFEAST_POSTGRES_DATABASE
- Execute the
initializestep in Valohai. It uploads the raw example data into a database tablefeast_driver_hourly_stats. - Execute the
applystep in Valohai. It uploads all the feature definitions into the database.