Submit User model to InferX platform - inferx-net/inferx GitHub Wiki

After deploy InferX platform with https://github.com/inferx-net/inferx/wiki/InferX-platform-0.0.1-deployment, we can submit user model to the platform.

Create API Key

  1. In the InferX dashbord, click "Apikey" and input username and password created in Keycloak. If deployment is based on Keycloak database image, the default username is "testuser1" and password is "test".
  2. In the Apikey page, input apikey name under textbox under "Add Apikey" and then click button "Add Apikey name".
  3. Then the new API Key will generated for the user

Set API Key

  1. Open local file path of https://github.com/inferx-net/inferx/blob/main/config/models.txt
  2. Edit IFERX_APIKEY with the generated apikey such as "97146956-2fd9-4b7e-8308-835d81dff85e"
  3. Open a bash terminal and copy and paste
export KEYCLOAK_URL="http://localhost:1260/authn"
export INFX_GATEWAY_URL="http://localhost:4000"
export IFERX_APIKEY="d485cf31-c313-426f-a3a8-116a7fa97d44"

Go to curent repo config folder

cd inferx/config

Create tenant: Copy and paste

/opt/inferx/bin/ixctl create public.json

Create namespace: Copy and paste

/opt/inferx/bin/ixctl create TinyLlama_namespace.json

Download model image from hugginface

In the example, TinyLlama/TinyLlama-1.1B-Chat-v1.0 is downloaded to "/home/brad/cache" folder, any there is one folder named /home/brad/cache/hub/models--TinyLlama--TinyLlama-1.1B-Chat-v1.0

Update TinyLlama-1.1B-Chat-v1.0.json

Please update the "mounts" ==> "hostpath" to your models folder. In the example, the folder is "/home/brad/cache".

Submit the model

/opt/inferx/bin/ixctl create TinyLlama-1.1B-Chat-v1.0.json

Snapshotting

When submitting is done, the InferX will start snapshoting. You can find a new pod created at Pods (http://[inferxhost]:81/listpod).

Snapshot ready

After snapshot is done, a new model in will appear in Models page (http://[inferxhost]:81/listfunc)