Tech Notes: Debugging LLVM + Rust

I’m working on a programming language, writing the compiler in rust. I’m stuck at this point from a segfault that occurs with the following IR (generated by my compiler):

; ModuleID = 'main'
source_filename = "main"

define void @main() {
  %result = call i64 @fib(i64 1)

define i64 @fib(i64) {
  %alloca = alloca i64
  store i64 %0, i64* %alloca
  %load = load i64, i64* %alloca
  switch i64 %load, label %switchcomplete [
    i64 0, label %case
    i64 1, label %case1

switchcomplete:                                   ; preds = %case1, %entry, %case
  %load2 = load i64, i64* %alloca
  %binop = sub i64 %load2, 1
  %result = call i64 @fib(i64 %binop)
  %load3 = load i64, i64* %alloca
  %binop4 = sub i64 %load3, 2
  %result5 = call i64 @fib(i64 %binop4)
  %binop6 = add i64 %result, %result5
  ret i64 %binop6

case:                                             ; preds = %entry
  ret i64 0
  br label %switchcomplete

case1:                                            ; preds = %entry
  ret i64 1
  br label %switchcomplete

This segfaults whenever I run my compiler, which currently compiles the code and immediately executed it in LLVM’s MCJIT.


Whenever I run my code in my debugger, I find that I have a segfault which doesn’t occur (at least at the same time) as when I run my app on the command line.

VS Code’s debugger returns:

so something is happening during the FPPassManager. Apparently the FPPassManager is what handles generating code for functions (read in the source code)

getNumSuccessors was a bit nebulous for me… what does this function actually do? I wasn’t familiar with the term “successor”: it must be something custom to LLVM. Some Googling finds:

So I guess successor is referring to the number of statements that immediately follow the existing statement. getNumSuccessors in core.h of llvm specifies there are function calls for a terminator. So what precisely is a terminator?

Looking through the LLVM source code again, it’s the classification for instructions that will terminate a BasicBlock. The list from LLVM9 looks like:

  /* Terminator Instructions */
  LLVMRet            = 1,
  LLVMBr             = 2,
  LLVMSwitch         = 3,
  LLVMIndirectBr     = 4,
  LLVMInvoke         = 5,
  /* removed 6 due to API changes */

Looking at the traceback, this is specifically occurring in the updatePostDominatedByUnreachable. The source code for that is:

/// Add \p BB to PostDominatedByUnreachable set if applicable.
BranchProbabilityInfo::updatePostDominatedByUnreachable(const BasicBlock *BB) {
  const Instruction *TI = BB->getTerminator();
  if (TI->getNumSuccessors() == 0) {
    if (isa<UnreachableInst>(TI) ||
        // If this block is terminated by a call to
        // @llvm.experimental.deoptimize then treat it like an unreachable since
        // the @llvm.experimental.deoptimize call is expected to practically
        // never execute.

The actual errors occurs on the first instruction of the assembly instruction:

; id = {0x00012806}, range = [0x000000000093fbb0-0x000000000093fc3b), name="llvm::TerminatorInst::getNumSuccessors() const", mangled="_ZNK4llvm14TerminatorInst16getNumSuccessorsEv"
; Source location: unknown
555555E93BB0: 0F B6 47 10                movzbl 0x10(%rdi), %eax
555555E93BB4: 48 8D 15 81 3B D5 01       leaq   0x1d53b81(%rip), %rdx
555555E93BBB: 83 E8 18                   subl   $0x

I can’t read assembler very well. But since this is a method, most likely the first instruction has to do with loading the current object into memory. Most likely then, getNumSuccessors is receiving a pointer to something it doesn’t expect. Most likely this is an NPE.

My hunch now is I have a basic block without a terminator statement, causing the JIT pass to fail.

There was a missing return statement on the main function. Adding that didn’t change anything.

Fixing the blocks to only have terminators did indeed fix the issue! Ultimately figuring out that a validator existed, and heeding it’s error messages lead to the solution.

Tech Notes: Updating Unity for Cerebrawl

I’m interested in starting a habit of note taking while I take on some pretty difficult tasks, maybe as a learning experience for myself or others if they find it valuable.

Today, I’ll be tackling Updating Cerebrawl’s Unity from 5.6 to 2018.3.

This is actually pretty late in the journey: I’ve got a branch of 2018.3 working, I just need to figure out how to reconcile that with the month-and-a-half’s worth of changes that were made in the meantime.

My upgrade path thus far has been a combination of the following tools:

  • vscode, when I need to go look at live coude
  • sourcetree, when I need to do some fine-grained change picking
  • Unity, to see if the things runs.

Errors Again

Pulling up my branch again, there’s errors around the lack of a TMP_PRO namespace. It seems that TextMeshProUGUI doesn’t exist for TextMeshPro 1.3. Something to look into later, but for now commenting that out should be fine.

Next ran into a duplicate tk2dSkin.dll. It looks like that now goes in the “tk2d” directory, rather than “TK2DROOT”. So just delete the old one.

Cherry-Picking the New Changes

We had to revert the 2018 unity changes previously. Last time I tried to merge in the master branch (I use git-svn so it’s effectively the SVN tree), git I think got confused because I reverted a bunch of the changes I had done, breaking everything and requiring me to apply those changes again.

This time, I should only pull in the changed made after that point. I created another branch to keep my working changes from being broken and lost in history when I merge in other changes.

I can use git cherry-pick to specifically pick up diffs in that version range:

git cherry-pick b813563…5646829

Ran into multiple errors cherry-picking. Resolution is to pick up incoming changes again and again (these are Unity asset files so not ones I needed to touch to update).

Once those were done, I switched back to the Unity editor, let it load again.

It Works!

Huzzah! For the most part everything has migrated over. The biggest challenge on this one was upgrading tk2d toolkit, which was broken by newer Unity versions.

Merging Changes In

I hit another snag trying to merge files in. Git svn attempted to rebase my changes on top of the existing branch, which doesn’t work really well as it tries to merge diffs again.

My best hope is to basically construct a changeset that is all of the changes I made on what’s in SVN today. To do so I run:

git svn fetch
git checkout master
git reset --hard git-svn 
git clean -xdf
git checkout feature/merge-unity-2018
git reset --soft master
git commit

Finally a git svn push and all the changes have been made!