Our Algorand Indexer is finally done
And it took a very, very long time.
Something we haven't seen documented about Algorand Indexers is that you kinda need an SSD to run them. NVMe, preferably. Also it should be about 2 TB.
Not that you can't do it without one - we did, we don't have 2TB-NVMe-SSD-on-a-server-money, but getting one up and running and in sync took a long time. This rant is about documenting the process.
We started our indexer on an Oracle Cloud Ampere VM which had a 750GB disk. This was around the end of November. After it synced a few million blocks s l o w l y, and we looked at howbigisalgorand.com and realized that... it wouldn't fit.
About that time we decided to get a dedicated server for this and have a nice spacious 2TB disk all for our indexer. Also around this time, a breaking change version of Algorand Indexer was released, requiring indexing from scratch. Huh. A good opportunity to start fresh with the latest version, then.
So we started fresh. We wrote a script to track the indexer progress, spitting out the rounds imported per minute, every ten minutes. It started off quickly, like this:
1643297480 Do 27. Jan 15:31:20 UTC 2022 6798333 +78961 7896/minute
1643298080 Do 27. Jan 15:41:20 UTC 2022 6869972 +71639 7163/minute
1643298680 Do 27. Jan 15:51:20 UTC 2022 6950663 +80691 8069/minute
1643299280 Do 27. Jan 16:01:20 UTC 2022 7020789 +70126 7012/minute
1643299880 Do 27. Jan 16:11:20 UTC 2022 7092770 +71981 7198/minute
1643300480 Do 27. Jan 16:21:20 UTC 2022 7179650 +86880 8688/minute
1643301080 Do 27. Jan 16:31:20 UTC 2022 7255644 +75994 7599/minute
1643301680 Do 27. Jan 16:41:20 UTC 2022 7332953 +77309 7730/minute
1643302280 Do 27. Jan 16:51:20 UTC 2022 7414456 +81503 8150/minute
1643302880 Do 27. Jan 17:01:20 UTC 2022 7490323 +75867 7586/minute
1643303480 Do 27. Jan 17:11:20 UTC 2022 7566987 +76664 7666/minute
1643304080 Do 27. Jan 17:21:20 UTC 2022 7655334 +88347 8834/minute
1643304680 Do 27. Jan 17:31:20 UTC 2022 7735944 +80610 8061/minute
8K a minute! sweet. But... as we drew nearer to modern day rounds, the transactions got more plentiful and more complicated, our database grew more bloated, and the speed decreased significantly:
1649845964 Mi 13. Apr 10:32:44 UTC 2022 20368608 +864 86/minute
1649846565 Mi 13. Apr 10:42:45 UTC 2022 20369550 +942 94/minute
1649847166 Mi 13. Apr 10:52:46 UTC 2022 20370491 +941 93/minute
1649847766 Mi 13. Apr 11:02:46 UTC 2022 20371471 +980 98/minute
1649848367 Mi 13. Apr 11:12:47 UTC 2022 20372320 +849 84/minute
1649848968 Mi 13. Apr 11:22:48 UTC 2022 20373281 +961 95/minute
1649849569 Mi 13. Apr 11:32:49 UTC 2022 20374394 +1113 111/minute
1649850169 Mi 13. Apr 11:42:49 UTC 2022 20375400 +1006 100/minute
1649850770 Mi 13. Apr 11:52:50 UTC 2022 20376592 +1192 119/minute
1649851372 Mi 13. Apr 12:02:52 UTC 2022 20377826 +1234 122/minute
1649851973 Mi 13. Apr 12:12:53 UTC 2022 20378920 +1094 109/minute
1649852574 Mi 13. Apr 12:22:54 UTC 2022 20379921 +1001 99/minute
1649853174 Mi 13. Apr 12:32:54 UTC 2022 20380920 +999 99/minute
1649853776 Mi 13. Apr 12:42:56 UTC 2022 20381585 +665 66/minute
1649854377 Mi 13. Apr 12:52:57 UTC 2022 20382239 +654 65/minute
1649854977 Mi 13. Apr 13:02:57 UTC 2022 20382550 +311 31/minute
1649855577 Mi 13. Apr 13:12:57 UTC 2022 20382686 +136 13/minute
That last log line, 13/minute? That's what you expect when you finally sync. After that we had line after line of glorious "13/minute".
The tables are big:
"txn" lifts the trophy at 905GB.
The choke point was definitely the disk. We had a beefy 8 core CPU which was mostly chilling and 32GB of RAM, which we ended up making the most of by tweaking postgres configs. Our sweet spot was shared_buffers=24GB
It took us from January 27th until April 13th to get this puppy up to date. 77 days.
Kids, if you're going to run an Algorand Indexer, we recommend being born rich and "just" getting a server with a 2TB NVMe SSD.
The rest of us need patience.