New · interactive
The Journey of One Token →

Three interactive pieces on how an LLM inference server actually works, from one token arriving at the load balancer to a byte leaving HBM — backed by measurements on GPUs I rented. Start here for the reading order.

The Journey of One Request

 Interactive · animated walkthrough

Follow a single request from the moment it hits the load balancer to the moment the last token comes back. Each of the 12 stops is a real mechanism — the queue, the prefill vs decode split, the KV cache being read, prefix caching, speculative decoding — with an inline glossary you can click for definitions on demand. [Read More]

Serving a Trillion Parameters

 Interactive · expandable diagram

Zoom the same question up to a model that doesn't fit on one card — or eight, or sixteen. Four nested levels: data center → nodes → GPUs → SM. Watch what happens to weights (they stay put), the KV cache (it never travels), and the one thing that does move — the activation — as it crosses NVLink, InfiniBand, and finally HBM. [Read More]

The Inference Engine, in Cross-Section

 Interactive · long-form with charts

The technical companion to the first two. Prefill (compute-bound) vs. decode (bandwidth-bound), continuous batching, PagedAttention, prefix caching, speculative decoding — each explained mechanically with a chart, then a results table at the bottom summarizing every measured speedup across the whole project. [Read More]

Sport fans' favor via SimRank

 Posted on June 2, 2021  |  2 minutes  |  362 words  |  Chenxin Ma

In the previous post, I played with SimRank algorithm on Arxiv data to show the potential relationship among research areas. In this post, I would like to apply the same thing on data from a Chinese sport forum, hupu. In general, the question is “is there a pattern for sport fans to support different teams? For example, a fan supports Real Madrid will rarely supports Barcelona at the same time”. To me, as a deep sport fan, the final question is “For what reasons, a fan becomes a fan for that team”? [Read More]
data mining  algorithm 

Symmetry of NN: Experiments of different directions in optimization for tiny neural nets

 Posted on December 18, 2020  |  5 minutes  |  905 words  |  Chenxin Ma

This is a proposal which fails to be turned into a paper during my PhD. Problem and Data Let’s only focus on two simple networks and two sets of one dimensional data. Network 1 (N1) has the 1−2−1 structure, thus has 7 parameters in total. Network 2 (N2) has the 1−1−1 structure, thus has 4 parameters in total. I tried linear, Relu, sigmoid as three activation functions and square loss as the loss function. [Read More]

Research connections via SimRank

 Posted on November 20, 2020  |  3 minutes  |  520 words  |  Chenxin Ma

I feel PageRank and SimRank are very interesting algorithms after learning them. They are able to provide fascinating insights for connections in any area. This blog is one of my attempts of applying SimRank to find such connections, in research areas. We all use Arxiv to post our research results. When I posted my first paper, there was a big question to me: How to choose the tags as areas of research? [Read More]
data mining  algorithm 

A benchmark of open source VRP solvers

 Posted on October 1, 2020  |  2 minutes  |  406 words  |  Chenxin Ma

1. Motivation of the benchmark Figure out how different algorithms in various solvers perform. 2. Benchmark cases Gehring & Homberger vrptw 200, 600 cases 3. Solver options solver initial solution algo heuristic # threads ortools Christofides (chris) gls, sa, gts* 1 ortools Path Cheapest Arc (pca) gls, sa, gts 1 ortools Savings gls, sa, gts 1 ortools ensemble** gls, sa, gts 1 jsprit default default 1, 2, 8 *Heuristic names: guided local search (gls), simulated annealing (sa), genetic tabu search (gts) [Read More]

Call Cuda code in Java Through JNI

 Posted on May 16, 2020  |  2 minutes  |  284 words  |  Chenxin Ma

Here is a step by step instruction of compiling cuda code through JNI interface. Declared a simple native method, and called System.loadLibrary in java package com.name.package.jniexample; public class HelloJNI { static { try{ System.loadLibrary("hello"); } catch(UnsatisfiedLinkError e){ System.err.println("Cannot load .so library.") e.toString(); } } private native void sayHello(String str); public static void main(String[] args) { new HelloJNI().sayHello(); } } Compile the java file, create header file javac $(dir)/HelloJNI. [Read More]

Cuda_combination_search

 Posted on April 3, 2020  |  3 minutes  |  491 words  |  Chenxin Ma

#include <stdio.h>#include <thrust/host_vector.h>#include <sys/time.h>#include <time.h> #define MAX_N 12 #define nTPB 256 #define GRIDSIZE (32*nTPB) #define cudaCheckErrors(msg) \ do { \ cudaError_t __err = cudaGetLastError(); \ if (__err != cudaSuccess) { \ fprintf(stderr, "Fatal error: %s (%s at %s:%d)\n", \ msg, cudaGetErrorString(__err), \ __FILE__, __LINE__); \ fprintf(stderr, "*** FAILED - ABORTING\n"); \ exit(1); \ } \ } while (0) // thrust code is to quickly prototype a CPU based // method for verification int increment(thrust::host_vector<unsigned> &data, unsigned max){ int pos = 0; int done = 0; int finished = 0; while(! [Read More]

Add scheduled jobs via Cron on AWS EC2

 Posted on April 2, 2020  |  1 minutes  |  149 words  |  Chenxin Ma

By using Cron, we can schedule the running of scripts at a specific date and time. However, it took me a while to figure out how to make it work on AWS EC2 to schedule a python task. Below are steps. 1). Need to add conda activate env_name in bash file, so that conda environment is activated automatically before Cron runs. 2). Create Cron jobs by crontab -e. After setting up the scheduled time following syntax of Crontab, we need to first cd to the path of python script. [Read More]
linux  memo