💾 Archived View for nox.im › posts › 2022 › 0207 › serum-decentralized-exchange-on-solana captured on 2023-09-28 at 16:04:45. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2023-04-26)
-=-=-=-=-=-=-
The Solana project Serum[1] is a model for decentralized exchanges with on-chain limit orderbooks and matching engines. The primary selling point of a DEX is permissionless, non-custodial Defi and the Solana blockchain brings speed and low transaction cost to the table. With block times of 400ms, it can currently process ~50,000 transactions per second. This allows Serum to handle hundreds of orders per second per market on-chain.
The success of **Serum** can be measured in the number of projects it has spawned since inception. In this post we're going through the basic setup of the Serum DEX and it's concepts. I found online material to be scattered and outdated at times and hope to bring some clarity and a tutorial to this space.
Let me know if you have any questions. What we're setting up looks like the following screenshot. Everything discussed here is open source. The end result shall look like the following screenshot.
Solana Serum DEX on localhost[1]
1: Solana Serum DEX on localhost
If you have not yet set up Rust and Solana, you can follow my post on the Solana development environment setup[1], start a local test validator and drop yourself some test SOL:
1: Solana development environment setup
solana-test-validator --no-bpf-jit solana airdrop 100
Git clone the serum-dex repository[1], build and deploy the rust program to the local test validator:
git clone https://github.com/project-serum/serum-dex.git cd serum-dex/dex cargo build-bpf solana deploy target/bpfel-unknown-unknown/release/serum_dex.so
After deployment, we get the program id that is relevant in addressing the program account going forward:
Program Id: HEKdDjSvXxxfDud8puPxgLYdWFfnVe8A4jENDxHzVR16
Note that the program is ~340K in size and will cost you ~2.42 SOL to deploy if this were mainnet.
ls -l target/bpfel-unknown-unknown/release/serum_dex.so -rwxr-xr-x 2 noxim users 348024 Jan 26 10:13 target/bpfel-unknown-unknown/release/serum_dex.so
We can estimate the cost for rent[1]:
solana rent 348024 Rent per byte-year: 0.00000348 SOL Rent per epoch: 0.006634331 SOL Rent-exempt minimum: 2.42313792 SOL
We now want to deploy a market and can use the crank program that the serum-dex ships.
We can deploy a sample exchange market with the `whole-shebang` crank program that setups up a coin mint, a pc mint, corresponding vaults and all other accounts required by a trading pair:
cd crank cargo run -- localnet whole-shebang ~/.config/solana/id.json HEKdDjSvXxxfDud8puPxgLYdWFfnVe8A4jENDxHzVR16
It takes about 15 seconds to run whole shebang and relevant output for us are these bits:
Coin mint: 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS
Pc mint: 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY
Vault signer nonce: 0
Creating coin vault...
Created account: HQzahqbSP2EUX99ZucekJxmTKQ6JpcrLkMsiYPsu1b9r ...
Creating pc vault...
Created account: H5ymt1Hp25BDd8h8b5XjtmJff2KJP6EVbyqZuUBKEHJJ ...
...
Listing 6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN ...
Market keys: MarketPubkeys {
market: 6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN,
req_q: DWL23Yf4R2pycobpLfNCr2bYsocT3cbMY2eMSJ6SZpJR,
event_q: AcK9jke263Hh9vxKYKu4YHHwwYHmgN3qDCFB25EMM6K7,
bids: 7djc71dGdLRh4gyfp1hfsxeftzB6wyxQN6UQ1tm9nTTu,
asks: DRitKgVT18C5yprofY3Qcy9xZpp4YByHotRsybxSAvc2,
coin_vault: HQzahqbSP2EUX99ZucekJxmTKQ6JpcrLkMsiYPsu1b9r,
pc_vault: H5ymt1Hp25BDd8h8b5XjtmJff2KJP6EVbyqZuUBKEHJJ,
vault_signer_key: 8xuXac585gbD7qzk1wkRRhX6qAuhhe748fW47eXUHayy,
}
The coin vault and price currency vault are PDA accounts that hold the base and quote currency balances. With the vault signer key being a PDA with access to these accounts. From the crank code, the default sizes are as follows:
MarketAccountSize = 376 ReqQueueAccountSize = 640 EventQueueAccountSize = 1 << 20 BidsAccountSize = 1 << 16 AsksAccountSize = 1 << 16
The market account holds metadata such as tick and lot sizes and is 376 bytes in size. The request queue maintains order placement and cancellation requests and is 640 bytes in size. The event queue reports the outputs from matched or filled orders and is 1 MiB in size. The bid and ask accounts created here are 65 KiB in size. Note that the size of these accounts makes creating a serum market expensive.
The event queue and the bid and asks slabs are holding an array of items. When the event queue is full, the DEX will no longer work as fills can no longer be processed. When bid and ask slabs are full, non competitive orders are getting booted from the radix tree. In the Serum code this is called critbit. Crit-Bit (or radix) trees are data structures that are space-optimized and fits the on-chain constraints. Each node that is the only child is merged with its parent.
The minimum size of entries in the event queue is 128, plus the variables in the struct. The minimum size for the bid and ask slabs is 256. You have to size these accounts to trade off attack surface to what you want to pay for rent.
To visualize what we just deployed onto our local chain, we can use the serum-dex-ui[1] from Github:
git clone https://github.com/project-serum/serum-dex-ui.git cd serum-dex-ui
Edit `src/pages/TradePage.tsx` and uncomment the following stub for quick setup without the TradingView private dependency:
// import { TVChartContainer } from '../components/TradingView';
// Use following stub for quick setup without the TradingView private dependency
function TVChartContainer() {
return <></>
}
If you're running a local test validator that isn't on your local machine, for example on a raspberry pi validator[1], you can make it available via `src/utils/connection.tsx` and add a connection url:
// ...
{ name: 'localnet', endpoint: 'http://127.0.0.1:8899', custom: false },
{ name: 'raspberry pi', endpoint: 'http://10.3.141.135:8899', custom: false },
];
And run the Serum DEX exchange UI locally, you will need `npm` and `yarn`:
yarn yarn start # OR yarn build
If you want to deploy it to your test server where you may share a test validator with your team, update the `package.json` `homepage` field.
"homepage": "http://10.3.141.135/dex-ui/",
Then copy the contents of the build directory to your server root under `/dex-ui`.
When visiting the UI on localhost or remote, add a custom market with the `+` sign with the listing address above, name the listing and your coin and pc token for the trading pair. I'm creating a hypothetical NOX token against USDC on an arbitrarily named market `nox.im`. In the real world it'd be something like `BTC/USD`.
Solana Serum - Add Custom Market[1]
1: Solana Serum - Add Custom Market
Which will redirect you to the market `http://localhost:3000/#/market/6saerH11Lqky4QfLUG3Papjsb1eesuEeMcyvMEMBcAWN`.
For development, we're using the same SPL token wallet[1] that I used with the raspberry pi before. In the wallet connector pick "Sollet Extension".
In order to trade we need to create ourselves associated token accounts for the coin and pc mint and mint ourselves some tokens. We can use the Solana toolchain to create the two local token accounts for testing as follows:
spl-token create-account 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS Creating account 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV spl-token create-account 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY Creating account 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb
And mint the two tokens:
spl-token mint 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS 1000000 Minting 1000000 tokens Token: 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS Recipient: 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV
spl-token mint 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY 1000000 Minting 1000000 tokens Token: 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY Recipient: 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb
Create the token in your Sollet browser extension with the token mint address
Sollet Extension - Add SPL Tokens[1]
1: Sollet Extension - Add SPL Tokens
And transfer some tokens to your browser extension wallet, currently everything belongs to your `~/.config/solana/cli/config.yml` identity:
spl-token transfer 8gEsYkoLBpNht5wg1oAy2i6WKPmiAm1ectgGUA8jgvwS 10000 BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn Transfer 10000 tokens Sender: 6SCLkaqrTJB56VirYYeMMP5SjBuerJqnEbh4tfxDpZoV Recipient: BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn Recipient associated token account: 8usSUzEioSQ7MNAiD6EeKY7PDrG7g7esFrPQcin3CXna
spl-token transfer 8RRPdsZtkbAatuAgMNPiuJ1MTd2rXtB1tpeQpSJhnfQY 10000 BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn Transfer 10000 tokens Sender: 8dFBGd4E253yXeFoSs6aHcVP4r1x5Q3rCexuhiZ6wWvb Recipient: BFrqAdfjBrB4KAnp7p4dgxUAhpNh54NTcAHZZDjwstqn Recipient associated token account: GZc7y3aPg6VRHozmGbwpiZc2JeRk8cjJEK457hNgbNjq
You can find these in the Solana explorer by their wallet or transaction hash.
Solana explorer[1]
And on our main account we can check our token balances.
Solana explorer - token balances[1]
1: Solana explorer - token balances
You should now be able to connect the Serum DEX to your local test validator, the market we created with the whole-shebang crank and your browser extension wallet on the same cluster. If that works we should be able to make orders with the minted tokens we transferred to ourselves that belong to the token mints of the serum market.
The project Serum DEX operates its orderbook in integers. This means that you have to specify lot sizes for markets, offsetting the decimal places of the asset traded. The increment of both tokens cannot be below 1 unit of each. This can be found in `dex/src/instruction.rs`:
pub struct InitializeMarketInstruction {
// In the matching engine, all prices and balances are integers.
// This only works if the smallest representable quantity of the coin
// is at least a few orders of magnitude larger than the smallest representable
// quantity of the price currency. The internal representation also relies on
// on the assumption that every order will have a (quantity x price) value that
// fits into a u64.
// ...
pub coin_lot_size: u64,
pub pc_lot_size: u64,
// ...
// ...
pub pc_dust_threshold: u64,
}
For each Serum market, we have to specify the coin lot size (base token), the pc (price currency) lot size (quote token) and the price currency dust threshold.
For example, 2.3300 lot of the EURUSD with a quote of 1.1400 means that you will pay 2.6562 USD to buy 1EUR. Lot sizes in Serum work by offsetting the token mints decimal places. We can understand the base lot size offsetting the mint decimals, the quote lot size as offsetting the base decimals.
mint / base lot size 1000000 / 1000 = 1000 -> 3 decimal places for lots base lot size / quote lot size 1000 / 1 = 1000 -> 3 decimal places for the price
To summarize
- mint 6 decimals, e.g. USDT
- base lot size 1000
- quote lot size 1
will result in
- lot decimals $1/1000 = 0.001$
- price decimals $1000/10^6 = 0.001$
Inversely we can also compute the lot sizes from the decimal places for size, price and quantity. For the same example, we desire 3 decimal places for size and quantity and 3 decimals for the price.
- base lot size = $10^6*0.001 = 1000$
- quote lot size = $10^6*0.001*0.001 = 1$
If the compute any of these as a fraction of 1 (e.g. 0.1), it means our mint decimals doesn't accommodate the desired decimals in the market. Remember that every order will have a (quantity x price) value required that fits into a `u64`.
Solana Project Serum EUR/USD market[1]
1: Solana Project Serum EUR/USD market
Understanding base and quote lot sizes, we can not trade into Serum. Assuming we have a token with a 6 decimal mint, e.g. USDC, 1 USD will equal 1000000 base units. To programmatically find out the right denomination we can verify as follows:
- price: 10^(base mint decimals - price decimals) = 10^3
- size: 10^(size decimals) = 10^3
- quantity = price * size, $1.14*10^3 * 2.33*10^3 = 2656200$ = 2.6562 USD
As usual with Solana programs, DEX errors are in hexadecimal. Do check the file serum-dex/blob/master/dex/src/error.rs#L78[1] when you encounter errors. For example 0x34 is InsufficientFunds and indicates you don't have sufficient tokens for a trade to be made.
1: serum-dex/blob/master/dex/src/error.rs#L78
Custom program errors when dealing with serum, can either be looked up in an enum, or for assertion errors indicate the file and line number as per this comment in `dex/src/error.rs`.
#[derive(Debug, IntoPrimitive, FromPrimitive, Clone, Copy, PartialEq, Eq)]
#[repr(u32)]
pub enum DexErrorCode {
InvalidMarketFlags = 0,
// ...
CoinVaultProgramId = 10,
// ...
Unknown = 1000,
// This contains the line number in the lower 16 bits,
// and the source file id in the upper 8 bits
#[num_enum(default)]
AssertionError,
}
As a side note to this code comment above, bit orders are ordered like `<upper><lower>`.
For example, custom program error `0x100067e` hints at line 1662 in `state.rs`, meaning the assertion on that line is failing. In this case it indicates that one of the dex accounts isn't properly sized or padded when creating a market.
check_assert_eq!(data.len() % 8, 4)?;
When you're subscribing to web sockets on serum accounts, make sure you use the `memcmp` filter at offset 0 for the byte string "serum". All serum related accounts, when initialized, start with this padding. It avoid you having to filter later on in your code.
In order to interact with a Serum DEX deployment, clients have to create an open orders account. It stores amounts deposited into serum, tracks open orders with client IDs and serum internal IDs.
It should be noted that the open orders account tracks fills with a delay. There is a serum crank to process the event queue. When the outputs of the event queue have been processed, the total and free amounts are updated. The total amounts track the amounts free and outstanding in orders. The free amounts track what can be settled to the clients wallet token accounts.
When settling funds in the dex, we are withdrawing the free amounts from the open orders account to our wallets token account. The settle funds instruction zeroes out the free amounts and subtracts from the open orders total amounts and the total market deposits.