Deploying AI Models on the Web with ONNX on the DC-ROMA RISC-V AI PC

Deploying AI models in ONNX (Open Neural Network Exchange) format is a key technique for enabling real-time interactive experiences on the web. This guide, centered around the Transformers.js library and the DeepSeek-R1 model, provides a detailed, step-by-step walkthrough on the DC-ROMA RISC-V AI PC — from environment setup to front-end interaction.

1. Environment Setup

Device Specifications for Running the Demo

roma@roma ~> fastfetch
                             ....              roma@roma
              .',:clooo:  .:looooo:.           ---------
           .;looooooooc  .oooooooooo'          OS: Ubuntu 24.04 LTS  v1.0.150194
        .;looooool:,''.  :ooooooooooc          Host: DeepComputing FML13V03
       ;looool;.         'oooooooooo,          Kernel: Linux 6.6.92-eic7x-2025.7
      ;clool'             .cooooooc.  ,,       Uptime: 36 mins
         ...                ......  .:oo,      Packages: 2362 (dpkg)
  .;clol:,.                        .loooo'     Shell: fish 3.7.0
 :ooooooooo,                        'ooool     Display (BOE 13.3"): 2256x1504 @]
'ooooooooooo.                        loooo.    DE: GNOME 46.0
'ooooooooool                         coooo.    WM: Mutter (Wayland)
 ,loooooooc.                        .loooo.    WM Theme: Yaru
   .,;;;'.                          ;ooooc     Theme: Yaru [GTK2/3/4]
       ...                         ,ooool.     Icons: Yaru [GTK2/3/4]
    .cooooc.              ..',,'.  .cooo.      Font: Cantarell (11pt) [GTK2/3/4]
      ;ooooo:.           ;oooooooc.  :l.       Cursor: Yaru (24px)
       .coooooc,..      coooooooooo.           Terminal: /dev/pts/3
         .:ooooooolc:. .ooooooooooo'           CPU: hifive-unmatched-a00 (8) @ z
           .':loooooo;  ,oooooooooc            GPU: Img gpu [Integrated]
               ..';::c'  .;loooo:'             Memory: 2.29 GiB / 14.34 GiB (16)
                                               Swap: 256.00 KiB / 2.00 GiB (0%)
                                               Disk (/): 39.08 GiB / 56.28 GiB 
                                               Local IP (wlan0): 192.168.50.38/4
                                               Battery (Framework Laptop): 32% ]
                                               Locale: en_US.UTF-8
Code language: PHP (php)

System Environment Preparation

# nodejs
$ sudo apt install nodejs

# Check npm
$ npm --version
# If npm not found:
$ sudo apt install npm

# To run the demo, do not open the HTML file directly via double-click.
# If no preview command (e.g. yarn watch) is provided, start a simple HTTP server:
$ python3 -m http.server 8000
# localhost:8000/

# Or use light-server:
$ cnpm install light-server -g

## Start http server
$ ./node_modules/.bin/light-server -s . -p 8081
# localhost:8081
Code language: PHP (php)

Note: Clear your browser cache before reopening the demo to avoid errors.

2. ONNX

2.1 Using Transformers.js

Transformers.js uses onnxruntime as its backend. It provides a unified API for running different models and tasks, simplifying large model deployment. Reference: huggingface.co

If you need to use a non-SIMD version of ONNX, you’ll need to build it manually. The demo below includes this step, but you can try it yourself if needed.

2.1.1 Build Steps

1. Clone the repo:

$ git clone https://github.com/huggingface/transformers.js.git

$ cd transformers.js

# Remove package-lock.json to avoid version lock
$ rm package-lock.json
Code language: PHP (php)

2. Edit package.json to set onnxruntime-web version below 1.19.0:

"dependencies": {
    "onnxruntime-web": "1.18.0",
},
Code language: JavaScript (javascript)

Then modify webpack.config.js:

const ORT_JSEP_FILE = 'ort-wasm-simd-threaded.jsep.wasm';Code language: JavaScript (javascript)

3. Build

$ npm install          ## generate node_modules
$ npm run build        ## generate dist
Code language: PHP (php)

4. Copy

$ cp node_modules/onnxruntime-web/dist/*.wasm ./dist/

$ cp -r dist/  your_lib_path/transformers/dist/
# e.g. webai_test/onnx_deepseek/transformers

2.2 Setting Different Backends

Available execution providers in ONNX Runtime Web:

• 'wasm' – Default CPU backend
• 'webgpu' – WebGPU backend
• 'webnn' – WebNN backend
• 'webgl' – WebGL backend

2.2.1 Example with Transformers.js:

generator = await pipeline(
            ...
            {
                device: "wasm",
            },
        );
Code language: JavaScript (javascript)

2.2.2 Example with ONNX directly

1. Update your import statement(e.g webGPU):

For HTML script tag, change ort.min.js to ort.webgpu.min.js:

<script src="https://example.com/path/ort.webgpu.min.js"></script>Code language: HTML, XML (xml)

For JavaScript import statement, change onnxruntime-web to onnxruntime-web/webgpu:

import * as ort from 'onnxruntime-web/webgpu';Code language: JavaScript (javascript)

2. See Conditional Importing for details.

const mySession = await ort.InferenceSession.create(modelUrl, {..., executionProviders: ['webgpu', 'wasm']});Code language: JavaScript (javascript)

3. Demo Example

3.1 Project Structure

roma@roma ~/w/onnx_deepseek> tree
.
├── DeepSeek-R1-Distill-Qwen-1.5B-ONNX
│   ├── config.json
│   ├── generation_config.json
│   ├── onnx
│   │   ├── model_bnb4.onnx
│   │   ├── model_fp16.onnx
│   │   ├── model_fp16.onnx_data
│   │   ├── model_int8.onnx
│   │   ├── model.onnx
│   │   ├── model.onnx_data
│   │   ├── model_q4f16.onnx
│   │   ├── model_q4.onnx
│   │   ├── model_quantized.onnx
│   │   └── model_uint8.onnx
│   ├── README.md
│   ├── special_tokens_map.json
│   ├── tokenizer_config.json
│   └── tokenizer.json
├── index.html
├── README.md
└── transformers
    └── dist
        ├── ort-training-wasm-simd.wasm
        ├── ort-wasm-simd.jsep.wasm
        ├── ort-wasm-simd-threaded.jsep.wasm
        ├── ort-wasm-simd-threaded.wasm
        ├── ort-wasm-simd.wasm
        ├── ort-wasm-threaded.wasm
        ├── ort-wasm.wasm
        ├── transformers.js
        ├── transformers.js.map
        ├── transformers.min.js
        ├── transformers.min.js.map
        ├── transformers.node.cjs
        ├── transformers.node.cjs.map
        ├── transformers.node.min.cjs
        ├── transformers.node.min.cjs.map
        ├── transformers.node.min.mjs
        ├── transformers.node.min.mjs.map
        ├── transformers.node.mjs
        ├── transformers.node.mjs.map
        ├── transformers.web.js
        ├── transformers.web.js.map
        ├── transformers.web.min.js
        └── transformers.web.min.js.map

5 directories, 40 files
Code language: JavaScript (javascript)

3.2 Download DeepSeek model

git clone https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNXCode language: PHP (php)

3.3 Transformers.js Files

See section 2.1 Build Steps above.

3.4 index.html Example

A full front-end implementation using Transformers.js and the DeepSeek-R1-Distill-Qwen-1.5B-ONNX model — featuring a modern chat UI, real-time status display, and smooth model initialization.

Please refer to this document for the demo code.

https://drive.google.com/file/d/1IgUolyQyj5j4t2EWMy2_U-aik336JV-W/view?usp=sharing

Github link: https://github.com/DC-Yanwei/Web-AI-on-RISC-V.git

3.5 Run the Demo

# Start a lightweight local HTTP file server
roma@roma ~/w/onnx_deepseek> python3 -m http.server 8000

# Open your browser and visit: localhost:8000/
Code language: PHP (php)

Note: When opening a page, remember to clear your browser cache — otherwise, unexpected errors may occur.

For Chrome Browser:

1. Open Chrome Dev version and enter about://flags in the address bar.
2. Search for “webgpu” and set Unsafe WebGPU to Enabled.

## You can test with the following code
console.log(navigator.gpu)Code language: PHP (php)

3.6 Results

Notes

If you see an error saying the model failed to load, you can manually download the model model_unit8.onnx from the following link: https://huggingface.co/onnx-community/DeepSeek-R1-Distill-Qwen-1.5B-ONNX/tree/main/onnx Then copy it to the directory:

./webai_test/onnx_deepseek/DeepSeek-R1-Distill-Qwen-1.5B-ONNX/onnx

Known Issues

1. Web LLM currently runs only on the CPU — even with WebAssembly (WASM), AI response speed is significantly slower.
2. Streaming output is not yet supported; responses appear only after the AI finishes generating them.
3. The frontend currently lacks progress indicators for backend processing.

References

Web AI Setup Examples:

1. TensorFlow.js Setup Tutorial
2. ml5.js Documentation
3. ONNX Runtime Web Tutorial