Reproducing the issue​

I was actually asked about this occurring in the Speedrun Practice Mod when pause buffering the top of Citadel power cell. Under the right circumstances, in the pause menu when you press Circle to switch to the Options page, it will transition out of the power cell page, but never actually transition into the Options page. You can't unpause or go back in the menu - the game appears completely frozen!

So what's the exact setup that we need to reproduce? First we need a fresh save file with no previous autosaves, so that the next autosave triggers the popup explaining autosave and the memory card icon.

Then we need a way to trigger both the pause menu and the autosave popup simultaneously. Normally if you try to pause when the autosave popup is about to come up, it will force a transition from the power cell page to the autosave one.

On certain power cells, we can use a pause buffer (pressing Start and L2/R2 on the same frame) before collecting the cell - this allows us to get credit for collecting the cell, but opens the pause menu before the power cell animation can start.

Importantly in this scenario, despite not playing the power cell animation, the autosave process is still kicked off. And if we do this on a power cell and trigger the first autosave on this file, we'll be met with the frozen screen.

Hardlock or softlock?​

For most crashes that happen on PS2, when reproduced in OpenGOAL, the program will fully crash, and if we attach either the GOAL or a C++ debugger we can catch an exception being thrown.

In the case of this autosave/pause menu freeze however, OpenGOAL stays open, and no exceptions are thrown - so is the game actually frozen?

We can actually attach the REPL to the game running in debug mode, and even during the freeze the game will be responsive to code that we run via the REPL. So the game IS still running, it's just stuck in some sort of loop where it can't move on and doesn't respond to any inputs.

Debugging the issue​

I started by looking at the state of the progress (pause menu) process while the game was frozen:

gc> (ppointer->process *progress-process*)
2124612        #x206b44              0.0000        #<progress progress suspended :state progress-normal :flags r   :stack 112/256 :heap 6608/6608 @ #x206b44>

We can see it's in the progress-normal state, so I poked around there, trying to see how it transitions between pages while in that state. I found this bit of code:

(when (and (not (handle->process (-> *game-info* auto-save-proc)))
           (or (-> self force-transition) (-> self in-transition))
           (>= (-> self transition-offset)
               (if (and (zero? (-> self level-transition))
                        (nonzero? (-> self next-display-state))
                        (!= (-> self next-display-state) 1)
                        (!= (-> self next-display-state) 2))
                  512
                  256)))
  (if (>= (the-as int (-> self next-display-state)) 0) (enter! self (-> self next-display-state) 0) (pop! self))

And looking at the enter! method we can see this is where the display-state (page) of the menu is updated:

(defmethod enter! ((this progress) (screen progress-screen) (option int))
  (when (!= (-> this display-state) screen)
    (set! (-> *progress-state* yes-no-choice) #f)
    (set! (-> this selected-option) #f)
    (set! (-> this option-index) option)
    (set! (-> this last-option-index-change) (-> *display* real-frame-counter))
    (set! (-> this display-state) screen)

We can use the REPL again to see the value of display-state and next-display-state in the progress process during the freeze:

gc> (-> (the progress (ppointer->process *progress-process*)) display-state)
0        #x0              0.0000        0

gc> (-> (the progress (ppointer->process *progress-process*)) next-display-state)
3        #x3              0.0000        #<invalid object #x3>

And checking this against the progress-screen enum, these values makes sense given where the freeze occurs:

(defenum progress-screen
  :type int64
  (invalid -1)
  (fuel-cell 0)
  (money 1)
  (buzzer 2)
  (settings 3)

So for some reason, we're never making that enter! call to switch over to the settings page. I used the REPL to check the conditions in the when block required to make it to the enter! call, and found only one was failing:

gc> (not (handle->process (-> *game-info* auto-save-proc)))
1375524        #x14fd24              0.0000        #f

gc>  (handle->process (-> *game-info* auto-save-proc))
2142564        #x20b164              0.0000        #<auto-save auto-save suspended :state done :flags r   :stack 96/512 :heap 1088/1088 @ #x20b164>

So as expected, this is definitely autosave related. Now we need to figure out why the auto-save process is getting stuck in the done state.

(defstate done (auto-save)
  :virtual #t
  :code
    (behavior ()
      ...
          (when (= (-> *game-info* auto-save-count) 1)
           (set! (-> self event-hook) (-> (method-of-object self error) event))
           (set! (-> self state-time) (-> *display* real-frame-counter))
           (while (< (- (-> *display* real-frame-counter) (-> self state-time)) (seconds 0.2))
             (if (not (progress-allowed?)) (set! (-> self state-time) (-> *display* real-frame-counter)))
             (suspend))
           ;; Don't display the auto-save warning prompt if speedrunning
           (when (= (-> *pc-settings* speedrunner-mode?) #f)
             (activate-progress *dproc* (progress-screen auto-save))))))))

So normally here the auto-save process (if it's our first time and we're not in speedrun mode) tries to activate progress and show the autosave/memory card warning page. It has a while loop which only runs if it's been less than 0.2s since the last time state-time was updated (when it reached done state).

This loop exists primarly to check progress-allowed? - if we're not allowed to open the progress menu right now (e.g. maybe we're in a cutscene), the auto-save's state-time is updated to the current time, and we suspend until the next frame, where we run these checks again.

(defun progress-allowed? ()
  (not (or (-> *setting-control* current talking)
           (-> *setting-control* current movie)
           (movie?)
           (handle->process (-> *game-info* pov-camera-handle))
           (handle->process (-> *game-info* other-camera-handle))
           (< (-> *display* base-frame-counter) (-> *game-info* letterbox-time))
           (< (-> *display* base-frame-counter) (-> *game-info* blackout-time))
           (!= (-> *setting-control* current bg-a) 0.0)
           (!= (-> *setting-control* current bg-a-force) 0.0)
           (not (-> *setting-control* current allow-progress))
           (or (and (handle->process (-> *game-info* auto-save-proc))
                    (not (send-event (handle->process (-> *game-info* auto-save-proc)) 'progress-allowed?)))
               (not *target*)
               (= *cheat-mode* 'camera)))))

Again, we can go back to the REPL during the freeze and figure out what's failing here:

gc> (-> *setting-control* current movie)
1921416        #x1d5188              0.0000        240177

gc> (movie?)
1375532        #x14fd2c              0.0000        #t

gc> (handle->process (-> *game-info* other-camera-handle))
2507652        #x264384              0.0000        #<othercam othercam suspended :state othercam-running :flags     :stack 112/256 :heap 480/480 @ #x264384>

Because the power cell animation was queued up to be played after touching the cell, the game actually considers us in a cutscene, causing progress-allowed? to return #f.

Now we can see the full picture - because we were able to trigger the autosave and bring up the pause menu simultaneously, each process is stuck waiting on conditions that will never happen! The auto-save can't proceed because progress-allowed? is false due to the pending cutscene. But the pending cutscene can't play out (unblocking progress-allowed?) because we're stuck in the pause menu - and that's because progress won't transition while there's an active auto-save process.

Funny enough, using the GOAL debugger we can (:break) during the freeze to force a breakpoint in the code, wait a bit, and then (:cont) to resume from the breakpoint, and we'll artificially satisfy the state-time check, and un-freeze the game!

The fix​

The "proper" fix for this would probably be to prevent cells from being picked up in the pause buffer scenario, but changing that would be a bit invasive and too much divergence from original behavior (it would also break some speedrun strategies).

Since the issue is not something casual players are likely to run into, we can just focus on fixing this for speedrunners. As we saw earlier, there's already some code added in OpenGOAL that prevents the autosave popup from being activated if we're in speedrun mode. We can simply move the check up a bit earlier, and avoid reaching the code checking state-time and progress-allowed?:

-    (when (= (-> *game-info* auto-save-count) 1)
+    ;; og:preserve-this Don't display the first auto-save warning prompt if speedrunning
+    (when (and (= (-> *game-info* auto-save-count) 1) (not (-> *pc-settings* speedrunner-mode?)))
      (set! (-> self event-hook) (-> (method-of-object self error) event))
      (set! (-> self state-time) (-> *display* real-frame-counter))
      (while (< (- (-> *display* real-frame-counter) (-> self state-time)) (seconds 0.2))
        (if (not (progress-allowed?)) (set! (-> self state-time) (-> *display* real-frame-counter)))
        (suspend))
-      ;; Don't display the auto-save warning prompt if speedrunning
-      (when (= (-> *pc-settings* speedrunner-mode?) #f)
-        (activate-progress *dproc* (progress-screen auto-save))))))))
+      (activate-progress *dproc* (progress-screen auto-save)))))))

Now the auto-save process will finish executing and be deactivated successfully - unblocking the progress code and allowing us to call enter! and access the Options page!

References​

• PR #4340 (the fix for speedrun mode)

Pinpointing the crash​

First let's see if we can catch exactly where the crash happens by attaching the debugger. In the REPL, after we've compiled (ran (mi)) and launched the game in debug mode, we can run (lt) to connect the REPL, and then (dbgc) to connect the GOAL debugger.

Now we just need to reproduce the crash, and in the REPL we'll see it catches the exception:

Target raised an exception (EXCEPTION_ACCESS_VIOLATION [0xC0000005]). Run (:di) to get more information.

Running (:di) will show us the stack trace:

In function (method get-height-at-point xz-height-map) in segment 0 of obj height-map, offset_obj 0x817, offset_func 0xf3

height-map.gc:99
              (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
                                                 ^
  [0x18D4BE6AA24] mov rdi, r12                               mov igpr-48, igpr-24
  [0x18D4BE6AA27] mov rsi, r10                               mov igpr-49, igpr-45
  [0x18D4BE6AA2A] add r9, r15                                call igpr-50 (ret igpr-47) (args igpr-48 igpr-49)
  [0x18D4BE6AA2D] call r9

height-map.gc:99
              (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
                                              ^
  [0x18D4BE6AA30] imul eax, ebp                              imul igpr-52, igpr-32
  [0x18D4BE6AA33] movsxd rax, eax

height-map.gc:99
              (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
                    ^
  [0x18D4BE6AA36] add r11, rax                               addi igpr-43, igpr-52
  [0x18D4BE6AA39] xor r9, r9                                 mov-ic igpr-53, 0
  [0x18D4BE6AA3C] add r11, r9                                addi igpr-43, igpr-53

height-map.gc:99
              (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
                 ^
  [0x18D4BE6AA3F] mov r9d, [r15+rbx*1+0x1C]                  mov igpr-55, [igpr-3 + 28]
  [0x18D4BE6AA44] add r11, r9                                addi igpr-54, igpr-57

height-map.gc:102
          (f3-3 (the float (-> a1-7 0)))
                   ^
- [0x18D4BE6AA47] movsx r9, byte ptr [r15+r11*1] mov igpr-61, [igpr-59 + 0]
  [0x18D4BE6AA4C] cvtsi2ss xmm7, r9d                         i2f ifpr-60, igpr-61

height-map.gc:103
          (f4-1 (the float (-> a1-7 1)))
                   ^
  [0x18D4BE6AA51] movsx r9, byte ptr [r15+r11*1+0x01]        mov igpr-64, [igpr-59 + 1]
  [0x18D4BE6AA57] cvtsi2ss xmm6, r9d                         i2f ifpr-63, igpr-64

height-map.gc:104
          (f2-8 (the float (-> a1-7 v1-0)))
                              ^
  [0x18D4BE6AA5C] mov r9, rbp                                mov igpr-66, igpr-32

height-map.gc:104
          (f2-8 (the float (-> a1-7 v1-0)))
                   ^
  [0x18D4BE6AA5F] add r9, r11                                addi igpr-69, igpr-59
  [0x18D4BE6AA62] movsx r9, byte ptr [r15+r9*1]              mov igpr-68, [igpr-69 + 0]
  [0x18D4BE6AA67] cvtsi2ss xmm5, r9d                         i2f ifpr-67, igpr-68

rax: 0x0000000000000000 rcx: 0x0000000000147d21 rdx: 0x000000003c888889 rbx: 0x00000000022dd8f0 
rsp: 0x0000018d49d0ef50 rbp: 0x0000000000000050 rsi: 0x0000000000000000 rdi: 0x0000000000000000 
 r8: 0xfffffffffffffffe  r9: 0x00000000022db5f0 r10: 0x000000000000006e r11: 0xffffffff822db5f0 
r12: 0x0000000000000000 r13: 0x000000000028f884 r14: 0x0000000000147d21 r15: 0x0000018d49b90000 
rip: 0x0000018d4be6aa47

These stack traces can seem intimidating at first, but we really just want to focus on the code one line above where the debugger shows the breakpoint (the line in red):

height-map.gc:99
              (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
                 ^

Let's look at this in the source code for a bit more context:

(defmethod get-height-at-point ((this xz-height-map) (arg0 vector))
  (let* ((f0-1 (fmax 0.0 (* (-> this x-inv-spacing) (- (-> arg0 x) (-> this x-offset)))))
         (f2-4 (fmax 0.0 (* (-> this z-inv-spacing) (- (-> arg0 z) (-> this z-offset)))))
         (a2-0 (the int f0-1))
         (a1-1 (the int f2-4))
         (f1-7 (- f0-1 (the float a2-0)))
         (f0-4 (- f2-4 (the float a1-1)))
         (v1-0 (-> this x-dim))
         (a3-0 (-> this z-dim))
         (a1-7
           (the-as
             (pointer int8)
             (+ (+ (min a2-0 (+ v1-0 -2)) (* (min a1-1 (+ a3-0 -2)) v1-0) 0) (the-as int (the-as pointer (-> this data))))
             )
           )

It's a bit hard to read with the generated variable names, but this is computing a1-7 as a pointer to some address relative to (-> this data) as a starting point. Presumably the crash is caused by this pointing to an invalid memory address under some scenarios.

Making sense of xz-height-map​

What exactly is this in this context, and what's in its data field?

Well, we're in the get-height-at-point method for an xz-height-map, so we can look at that type's definition:

(deftype xz-height-map (structure)
  ((offset         float  3)
   (x-offset       float  :overlay-at (-> offset 0))
   (y-offset       float  :overlay-at (-> offset 1))
   (z-offset       float  :overlay-at (-> offset 2))
   (x-inv-spacing  float)
   (z-inv-spacing  float)
   (y-scale        float)
   (dim            int16  2)
   (x-dim          int16  :overlay-at (-> dim 0))
   (z-dim          int16  :overlay-at (-> dim 1))
   (data           (pointer int8))
   )

Not the most helpful... where is this actually used, and for what purpose? Let's peek at traffic-height-map.gc where I found the type referenced:

(define *traffic-height-map* (new 'static 'xz-height-map
                               :offset (new 'static 'array float 3 -3178496.0 71680.0 -3178496.0)
                               :x-inv-spacing 0.000010172526
                               :z-inv-spacing 0.000010172526
                               :y-scale 4096.0
                               :dim (new 'static 'array int16 2 80 #x70)
                               :data (new 'static 'array int8 8960
                                 ;; ~9000 lines of integers here, redacted
                                 )
                               )
        )

So data is a giant list of integers, and while it's still not immediately clear what this is, if we think about the names xz-height-map and get-height-at-point, and look at the implementation it begins to make a bit more sense. We can update the variable names and linebreaks to make it a bit easier to read:

(defmethod get-height-at-point ((this xz-height-map) (arg0 vector))
  (let* ((x-rel (fmax 0.0 (* (-> this x-inv-spacing) (- (-> arg0 x) (-> this x-offset)))))
         (z-rel (fmax 0.0 (* (-> this z-inv-spacing) (- (-> arg0 z) (-> this z-offset)))))
         (x-int (the int x-rel))
         (z-int (the int z-rel))
         (x-frac (- x-rel (the float x-int)))
         (z-frac (- z-rel (the float z-int)))
         (x-dim (-> this x-dim))
         (z-dim (-> this z-dim))
         (a1-7
           (the-as
             (pointer int8)
             (+ 
                (+ (min x-int 
                        (+ x-dim -2))
                   (* (min z-int 
                           (+ z-dim -2))
                      x-dim) 
                   0)
                (the-as int (the-as pointer (-> this data))))
             )
           )

The get-height-at-point method takes in a vector, and we can see the method only really cares about its X and Z coordinates. It uses the x-offset/z-offset, x-inv-spacing/z-inv-spacing, and x-dim/z-dim values to efffectively convert the coordinates into integer "buckets", and points to some offset from data using these values.

In other words, data is a 2D table of values, and it's looking up the value from this table based on the X and Z coordinates. The values represent the height at which zoomers should float, as this height varies throughout Haven City - you can sort of think of the data table like a pixelated version of a topography map:

So when does it crash?​

Now that we have a basic understanding of what the get-height-at-point method is doing, we can try to break down what happens leading up to a crash here. We can print some debug logs to check the value of these variables - in this example, I'm interested in the value of a2-0/x-int and a1-1/z-int in normal situations vs during the crash.

There are dozens of ways to print this value, but here's one quick and dirty solution to print it just before evaluating the pointer address:

(defmethod get-height-at-point ((this xz-height-map) (arg0 vector))
  (let* (...
         (a1-7
+          (begin
+           (format 0 "x-int: ~D / z-int: ~D~%" x-int z-int)
            (the-as
              (pointer int8)
              (+ 
                  (+ (min x-int 
                          (+ x-dim -2))
                    (* (min z-int 
                            (+ z-dim -2))
                        x-dim) 
                    0)
                  (the-as int (the-as pointer (-> this data))))
              )
+           )
           )

With this, we'll discover that in normal situations these two values are non-negative integers, and leading up to a crash one or both of them is a negative integer!

You might see where this is going - when we index into the data table using these values, the negative values will actually cause us to look outside of the data table area of memory, and thus cause a crash.

How do we fix it?​

Array out-of-bounds access is a classic bug when it comes to programming, and Naughty Dog attempted to account for it in this function.

The min calls here ensure that extra large values for x-int/z-int don't lead to array out-of-bounds access, capping them at x-dim/z-dim minus 2. My guess is that dim is short for dimension here, i.e. the length and width of the 2D array.

However it seems ND didn't consider the possibility of extra small X/Z coordinates, resulting in negative indexing. And that's precisely what happens when SFHBG goes overboard and sends Jak and the zoomer to extreme coordinates (in a certain direction).

The fix is straightforward, mirroring what ND did but on the other end of the spectrum - we just need to ensure we never use an index smaller than 0:

         (a1-7
           (the-as
             (pointer int8)
-             (+ (+ (min x-int (+ x-dim -2)) (* (min z-int (+ z-dim -2)) x-dim) 0) (the-as int (the-as pointer (-> this data))))
+             ;; og:preserve-this prevent array out-of-bounds access
+             (+ (+ (min (max 0 x-int) (+ x-dim -2)) (* (min (max 0 z-int) (+ z-dim -2)) x-dim) 0) (the-as int (the-as pointer (-> this data))))
             )
           )

That's it! Now we can let SFHBG rip without worry of crashing (though it can still sometimes lead to other glitches, like ending up stuck in a zoomer with a civilian driving).

This same bug exists in Jak 2 as well, and although we don't have the ability to do SFHBG in Jak 2 and test it out, we can make the same changes to ensure no array out-of-bounds access happens.

References​

• PR #4106 (the fix)

Multiple controllers​

Online multiplayer obviously comes with the additional challenge of building out the networking and synchronization systems, so in the beginning we focused on local multiplayer, with everything running in a single game instance.

So the first thing we needed to get working correctly was handling inputs from multiple controllers. GOAL already supported reading inputs from the 2nd controller actually, as there are some cheat codes and debug tools that read from it.

For example, in cheat mode if you press X on the 2nd controller during the rats mini-game (cpad-pressed? 1 x) (index 1 because of zero-indexing), it will automatically end the game:

(defstate billy-playing (billy)
  ...
  :code
    (behavior ()
      (loop
        (billy-game-update)
        (if (or (zero? (-> self num-snacks))
                (and (zero? (-> self num-rats)) (-> self passed-last-stage))
                (and *cheat-mode* (cpad-pressed? 1 x)))
          (go billy-done))

However despite the PS2 (with Multitap) supporting up to 4 controllers, any inputs on the 3rd or 4th controllers wouldn't be recognized in (Open)GOAL. We were hoping to support at least 4 players locally (I was envisioning a 4-way split-screen view), so we had some work to do here.

On the GOAL side of the code, the controllers are managed via the *cpad-list* which has an array of cpad-infos. We just needed to bump the size of this array from 2 up to 4, and then adjust the asserts to line up with the new memory layout:

;; List of controllers. It always has 2 controllers.
(deftype cpad-list (basic)
  ((num-cpads int32       :offset-assert 4)
-   (cpads     cpad-info 2 :offset-assert 8)
+   (cpads     cpad-info 4 :offset-assert 8)  ;; modified from 2->4 for PC 4-pad support
   )
  (:methods
   (new (symbol type) _type_ 0)
   )
  :method-count-assert 9
-  :size-assert         #x10
+  :size-assert         #x18
-  :flag-assert         #x900000010
+  :flag-assert         #x900000018
  )

Then in the C++ code, there were some similar bits of code that assumed only 2 controllers, which we extended to support 4:

/* newpad.h */
-static constexpr int CONTROLLER_COUNT = 2;  // support 2 controllers.
+static constexpr int CONTROLLER_COUNT = 4;  // support 4 controllers.

...

/* newpad.cpp */
struct GamepadState {
-  int gamepad_idx[CONTROLLER_COUNT] = {-1, -1};
+  int gamepad_idx[CONTROLLER_COUNT] = {-1, -1, -1, -1};
+  bool glfw_joystick_used[GLFW_JOYSTICK_LAST + 1] = {false};
} g_gamepads;

...

void check_gamepads() {
  auto check_pad = [](int pad) {  // -> bool
    if (g_gamepads.gamepad_idx[pad] == -1) {
      for (int i = GLFW_JOYSTICK_1; i <= GLFW_JOYSTICK_LAST; i++) {
-        if (pad == 1 && i == g_gamepads.gamepad_idx[0])
+        if (g_gamepads.glfw_joystick_used[i]) {
          continue;
+        }
        if (glfwJoystickPresent(i) && glfwJoystickIsGamepad(i)) {
          g_gamepads.gamepad_idx[pad] = i;
-          lg::info("Using joystick {}: {}, {}", i, glfwGetJoystickName(i), glfwGetGamepadName(i));
+          g_gamepads.glfw_joystick_used[i] = true;
+          lg::info("Using joystick {} for pad {}: {}, {}", i, pad, glfwGetJoystickName(i),
+                   glfwGetGamepadName(i));
          break;
        }
      }
    } else if (!glfwJoystickPresent(g_gamepads.gamepad_idx[pad])) {
-      lg::info("Pad {} has been disconnected", pad);
+      lg::info("Pad {} / joystick {} has been disconnected", pad, g_gamepads.gamepad_idx[pad]);
+      g_gamepads.glfw_joystick_used[g_gamepads.gamepad_idx[pad]] = false;
      g_gamepads.gamepad_idx[pad] = -1;
      return false;
    }
    return true;  // pad already exists or was created
  };
-  if (check_pad(0))
-    check_pad(1);
-  else
-    g_gamepads.gamepad_idx[1] = -1;

+  for (int i = 0; i < CONTROLLER_COUNT; i++) {
+    check_pad(i);
+  }
}

...

void update_gamepads() {
  check_gamepads();

-  if (g_gamepads.gamepad_idx[0] == -1) {
-    clear_pad(0);
-    clear_pad(1);
-    return;
-  }

  ...

-  read_pad_state(0);
-
-  if (g_gamepads.gamepad_idx[1] != -1)
-    read_pad_state(1);
-  else
-    clear_pad(1);
+  for (int i = 0; i < CONTROLLER_COUNT; i++) {
+    if (g_gamepads.gamepad_idx[i] != -1)
+      read_pad_state(i);
+    else
+      clear_pad(i);
+  }
}

And voila, now you could successfully connect up to 4 controllers, and read their inputs. For example, to check if the X button was pressed on the 3rd controller: (cpad-pressed? 2 x) (2 because of zero-indexing). Or to check if R2 is being held on the 4th controller: (cpad-hold? 3 r2).

Multiple Jaks​

One of the next hurdles we needed to solve was simply being able to spawn multiple instances of target, the process used for Jak as a playable character.

water111 actually toyed around with this very early on in OpenGOAL's development (ironically, before water was rendering 😅):

In this case he was just spawning Jak's model as a viewer process, meaning it didn't have any of the target code attached to it, and just rendered the model in a particular animation.

Normally there is only one target, referenced by the global *target* variable, and spawned by the start function which is called when starting a new game, loading a save, respawning after dying, etc:

(defun start ((arg0 symbol) (arg1 continue-point))
  (set! (-> *level* border?) #f)
  (set! (-> *setting-control* default border-mode) #f)
  (stop arg0)
  (let ((v1-3 (process-spawn target :init init-target arg1 :from *target-dead-pool* :to *target-pool* :stack *kernel-dram-stack*)))
    (if v1-3 (set! *target* (the-as target (-> v1-3 0 self))) (set! *target* #f)))
  *target*)

In theory you could spawn multiple target processes properly by making multiple of these process-spawn calls, but with vanilla code you will immediately crash and see the following in the console logs: WARNING: target pair had to be allocated from the debug pool, because *target-dead-pool* was empty.

We need to increase the size of *target-dead-pool*, which was written with the assumption there would only ever be a single target spawned:

-(define *target-dead-pool* (new 'global 'dead-pool 1 (* 48 1024) '*target-dead-pool*))
+(define *target-dead-pool* (new 'global 'dead-pool 4 (* 48 1024) '*target-dead-pool*))

If we try again to spawn a second target, we'll get a slightly different crash and message: WARNING: sidekick pair had to be allocated from the debug pool, because *16k-dead-pool* was empty.

sidekick here refers to the process for Daxter - we need to make a similar change to the dead pool that he is spawned from:

-(define *16k-dead-pool* (new 'global 'dead-pool 1 (* 16 1024) '*16k-dead-pool*))
+(define *16k-dead-pool* (new 'global 'dead-pool 4 (* 16 1024) '*16k-dead-pool*))

(Later on we discovered we would also need to extend the *8k-dead-pool* and *4k-dead-pool* in the same way to be able to use the Zoomer, FlutFlut, etc)

Great, now we can spawn multiple targets, but by default they will all respect controller 1. This is because init-target always assigns (-> *cpad-list* 0) as the controller to be used across the target code:

(defbehavior init-target target ((arg0 continue-point))
  ...
  (set! (-> self control unknown-cpad-info00) (-> *cpad-list* cpads 0))

Local Multiplayer​

As a proof-of-concept, we can just update the controller assignment after spawning additional targets to read from one of the other controllers:

(let ((target2 (process-spawn target :init init-target (get-continue-by-name *game-info* "village1-hut") :from *target-dead-pool* :to *target-pool* :stack *kernel-dram-stack*)))
  (when target2
    (set! (-> (the-as target (-> target2 0 self)) control unknown-cpad-info00) (-> *cpad-list* cpads 1))))

The camera goes a little crazy when you spawn additional targets like this, but aside from that, we now had 2 Jaks running around together, controlled by 2 different controllers - multiplayer!

Now there were still a variety of limitations with this initial version. One obvious one is that everybody shares one view and the camera is focused on only one of the targets. Related to this, the game only allows for 2 levels to be loaded at a time. So even if we supported real split-screen, players wouldn't be able to venture off into separate levels unless we refactored the whole level loading system too. For this and various other reasons, we didn't address these issues, and instead focused on getting everything else working in a shared-view local multiplayer.

We quickly found a handful of different behaviors in the game that misbehaved. One common theme was an assumption in the code that there would only ever be one target. For example, if anyone died, it would kill everyone off and reload from the last checkpoint, only respawning the first player, *target*. Or when player 2 reached a FlutFlut transpad and pressed Circle to get on, it would always send the event to player 1, *target*, and they would mount FlutFlut instead.

We started to track of all the targets in a global list, and stored each target's index in a field within the target object itself. With this, we were able to solve many issues by passing around the index as needed, and looking up the correct target from our list, instead of always using *target* like the original code.

(define MAX_MULTIPLAYER_COUNT 4)
(define *remote-targets* (new 'global 'boxed-array handle MAX_MULTIPLAYER_COUNT))

(defun get-target ((idx int))
  (if (and (-> *remote-targets* idx) (nonzero? (-> *remote-targets* idx)) (handle->process (-> *remote-targets* idx)))
    (the target (handle->process (-> *remote-targets* idx)))
    (the-as target #f)))

A couple of problem areas that needed more debugging and code changes were handling edge grabs, and interactions with water volumes. Without getting into too much detail, the edge grab code relies on some collision checks which cache tons of data in global objects like *collide-work* and *collide-cache*. These were designed to only hold enough data for a single target, but since they were being shared by every spawned target, you would get weird behavior like this:

Similar to the list of targets, we extended these systems as needed to hold lists instead of single objects, so each target instance had it's own dedicated version. Then we just need to replace all the references to these objects to pull from the list instead, using the specific target's index.

We made a few other small accessibility changes I won't go into detail about here:

• adding a button combo to spawn in a target for your controller
• making each target a different color
• using a wider camera than the default view
• respawning far-off players back at player 1

And with that, we had a pretty stable shared-screen local multiplayer working!

Online Multiplayer​

The next step here was of course to get this online so you could play with people across the world.

Steam and Parsec actually have some nice remote play features where the host's screen is shared to other players, and the other player's controller inputs are mapped as additional controllers back on the host's computer. So these tools could immediately be used with the local multiplayer - see this old setup guide Zedb0t put together:

While this was cool, in terms of actual playability, the single camera view was still an incredibly limiting factor. Since we wanted to keep exploring online options, rather than going back to split-screen support, we started thinking about ways to have multiple OpenGOAL instances communicating with each other. This way each player would have their own game running "like normal" with the camera focused on their Jak.

One of the earliest iterations of this is a proof-of-concept Zedb0t built where each OpenGOAL instance posted player positions to a Discord server, and read back other player's positions from that server. Then in each game, the remote player's position would be updated, in this case using the Farmer as a stand-in model.

As you can hear Zedb0t and Ruh discussing in the video, the latency was pretty bad when using a Discord server as the "host" for these messages between OpenGOAL instances, but the idea was there.

For the next iteration, we built and hosted a small Python web server that would receive HTTP requests from OpenGOAL and track each connected player's position. We extended the local multiplayer code to periodically send the server our current position, and request the position of all the other players so that we could update them in our local game.

In this way, everyone had their own OpenGOAL instance running with the camera focused on them, but remote players would still be seen moving around in their game. While we were only syncing position/rotation and not properly animating remote players, they were still able to pick up Orbs or Cells if they were in a level that you had loaded locally.

The level loading issue meant there was still de-sync across games if players went to different levels, but overall it was becoming much more playable. While not perfect, the latency in this version was way better than the Discord-based approach. It was even decent enough that we built out a Hide and Seek / Tag gameplay mode!

Around the same time Zedb0t and I were playtesting this with the community, Dexz revealed that he had been working on his own multiplayer project for OpenGOAL, called TeamRuns! His project was a bit more focused on syncing collectables and game state (e.g. for co-op speedrunning, lockout races. etc), and in this early version you wouldn't actually see other players in your game, only the result of their actions:

Zedb0t and I quickly connected with Dexz to try to work on combining our two projects. TeamRuns used peer-to-peer networking, and had way better latency than our HTTP dedicated server approach since it relied on UDP instead of TCP/IP.

Based on our multiplayer changes, Dexz was able to make changes to his code to allow for spawning multiple targets. So he started including player positions in the payloads sent across the network, updating remote target positions based on these similar to what we were doing in our previous approach.

A few months later, and Dexz had taken this one step further - he was now syncing animations for remote players as well! It was incredible how much more "real" the multiplayer felt when remote players were punching and rolljumping through your world instead of just sliding and floating around 😅

And that's more or less how we arrived at the modern version of online Jak multiplayer!

TeamRuns has a bunch of cool game modes and built-in speedrunning leaderboards. It's since been featured at the European Speedrunner Assembly as well:

Big shoutout to Dexz for his hard work on TeamRuns, be sure to check it out and play with your friends!

References​

• PR #1751 - 4 controller support
• Curl Hide and Seek branch
• TeamRuns website
• TeamRuns repo

I had never heard of such a bug, so my first guess was that this was another high FPS bug. But the user said no, it happens at 60 FPS and on console as well. Plus the claim that it only happened after progressing far enough into the game, and that it persisted after a reboot was strange...

You can see in this example that the Dark Jak icon stops flashing at around 2:36:05. The Dark Eco meter stops flashing later on around the 4.5 hour mark, just before the boss fight in Mars Tomb.

Investigation​

I started poking around the HUD code, looking for the code that added the flashing effect. My guess was that it would be referencing some kind of time-frame in order to flash with a consistent frequency.

In Jak 2, there are two separate HUD elements involved - hud-health which includes the Dark Eco meter, and hud-dark-eco-symbol which handles the Jak / Dark Jak icon. Sure enough in the draw method for both of these, we can see the flashing effect is achieved by scaling the sprite color based on a sine function involving the game-clock:

(defmethod draw ((this hud-health))
  ...
  (let ((v1-12
          (+ (the int (* 127.0 (sin (* 182.04445 (the float (* (-> *display* game-clock frame-counter) 2)))))) 127)
          )
        )
    (set! (-> this sprites 1 color x) v1-12)
    (set! (-> this sprites 1 color y) v1-12)
    (set! (-> this sprites 1 color z) v1-12)
    )
...

(defmethod draw ((this hud-dark-eco-symbol))
  ...
       (let ((v1-31
               (+ (the int (* 15.0 (sin (* 182.04445 (the float (* (-> *display* game-clock frame-counter) 4)))))) 160)
               )
             )
         (set! (-> this sprites 0 color x) v1-31)
         (set! (-> this sprites 0 color y) v1-31)
         (set! (-> this sprites 0 color z) v1-31)
         )
       )

As I expected, there was nothing related to checking game progression in this code, so what's causing the bug? Let's look more closely at the math for the flash effect.

The (-> *display* game-clock frame-counter) tracks how long you've been playing the game on this save file (e.g. for the playtime display on the save/load screens). This just an integer counting the number of ticks (300 ticks / sec) since the game was started (technically it start at (seconds 1000) = 300000). This doesn't include time where the game is paused or in cutscenes.

The bug lies within the (sin (* 182.04445 (the float (* (-> *display* game-clock frame-counter) <multiplier>)))) calculation - it turns out GOAL's sin breaks down when you feed it really big float values (~2147483584.0 or bigger):

gc> (sin 2147483400.0)
-1127675244        #xffffffffbcc90a94             -0.0245        #<invalid object #xbcc90a94>

gc> (sin 2147483500.0)
-1136062860        #xffffffffbc490e74             -0.0122        #<invalid object #xbc490e74>

gc> (sin 2147483600.0)
0        #x0              0.0000        0

gc> (sin 2147483700.0)
0        #x0              0.0000        0

gc> (sin 2147483800.0)
0        #x0              0.0000        0

gc> (sin 2147483900.0)
0        #x0              0.0000        0

gc> (sin 9999999999.0)
0        #x0              0.0000        0

Once your (-> *display* game-clock frame-counter) gets large enough, sin starts to return 0.0 no matter what, effectively pausing the flash effect.

For example, the code for the Dark Jak icon has a <multipler> of 4, so it will stop flashing after approximately (2147483584.0 / (182.04445 * 4) - 300000) / (300 * 60 * 60) = 2.45 hours, which roughly matches the example video from earlier.

So the bug isn't related to game progression, but rather game playtime! The (-> *display* game-clock frame-counter) value is stored in your savefile, which explains why the bug persists even after rebooting and loading your savefile. This bug also affects the Light Eco meter in Jak 3.

The Fix​

Thankfully dass saved me some head scratching and pointed out that the (* 182.04445 is effectively the same as the degrees macro (65536 / 360 = 182.0444):

(defglobalconstant DEGREES_PER_ROT 65536.0)

;; this was deg in GOAL
(defmacro degrees (x)
  "Convert number to degrees unit.
   Will keep a constant float/int constant."
  (cond
    ((or (float? x) (integer? x))
     (* DEGREES_PER_ROT (/ (+ 0.0 x) 360.0))
     )
    (#t
     `(* (/ (the float ,x) 360.0)
         DEGREES_PER_ROT
         )
     )
    )
  )

So first, we can refactor the code to be a bit more readable by using the degrees macro:

(defmethod draw ((this hud-dark-eco-symbol))
  ...
       (let ((v1-31
-              (+ (the int (* 15.0 (sin (* 182.04445 (the float (* (-> *display* game-clock frame-counter) 4)))))) 160)
+              (+ (the int (* 15.0 (sin (degrees (* (-> *display* game-clock frame-counter) 4))))) 160)

Now we see the frame-counter (times the <multiplier>) is being converted to degrees before calling sin on it. Since the sine function fluctuates between -1 and 1 and back every 360 degrees, this results in the flash effect cycling every 360 / <multiplier> ticks.

Knowing this, we can modulo the value by 360 and still get the same result from sin:

(defmethod draw ((this hud-dark-eco-symbol))
  ...
       (let ((v1-31
-              (+ (the int (* 15.0 (sin (* 182.04445 (the float (* (-> *display* game-clock frame-counter) 4)))))) 160)
+              (+ (the int (* 15.0 (sin (degrees (mod (* (-> *display* game-clock frame-counter) 4) 360))))) 160)

By doing this, we ensure the value we pass to sin is between (-360.0, 360.0), and avoid any large float values that would cause it to break. Now no matter how long you play, i.e. how big (-> *display* game-clock frame-counter) gets, the sin call should function correctly, and the HUD elements will keep flashing correctly!

References​

• PR #4285

To get Jak's health down to 1 hit, players would often stand on the torch and take several hits. But they started to notice something weird - the cooldown time in between taking damage from the torch would sometimes be longer than the normal 2 seconds. After encountering the issue several times, theUsual figured out a particular sequence of actions in the speedrun route that would cause the cooldown time to increase (the Temple climb and 2 restart missions to break into Temple), though it wasn't clear exactly what was happening.

Eventally I realized this was something I had also seen happen to Blahpy in Jak 2, on the hot plates in the Palace Cables level. But I still wasn't sure what exactly triggered the bug in either game.

Investigation​

First I wanted to understand how the 2 second cooldown between Jak taking damage works. After some poking around I realized that pretty much all attacks to Jak go through target-attacked, which for many cases makes a call to target-log-attack, where I noticed a suspiciously named value: (-> *TARGET-bank* same-attack-invulnerable-timeout).

(defbehavior target-log-attack target ((arg0 attack-info) (arg1 symbol)) 
  ...
  (let ((a3-0 (the-as object (-> self attack-info-old)))) 
    (dotimes (a2-2 8) 
      (let ((v1-9 (-> self attack-info-old a2-2))) 
        (when (= (-> arg0 id) (-> v1-9 id)) 
          (if (not (time-elapsed? (-> v1-9 attack-time) (-> *TARGET-bank* same-attack-invulnerable-timeout))) 
              (return #f) 
              ) 

Let's break this down - target (Jak) has an attack-info-old array of length 8, which we loop through and check if the old attack has the same id as the new attack we're processing. If there's a match, then we check if a certain amount of time same-attack-invulnerable-timeout has passed since the old attack's attack-time - if not, we're still in the cooldown window, so we return #f and the attack does nothing to Jak.

The same-attack-invulnerable-timeout is never changed anywhere in the code, it's always (seconds 2), so how could we end up getting longer cooldown windows? I started with my favorite brute force approach when I'm confused about the code / state of the game - printing a bunch of info to the console to verify my assumptions and hopefully find a lead.

Right before the line in question, I printed the value of (-> v1-9 attack-time), as well as (current-time) which is what the time-elapsed? macro compares against. I should mention that these fields are time-frames, which are really just integers counting the number of ticks (300 ticks / second) since some starting time, so they get printed as integers.

Under normal circumstances while standing on a torch, it would print time-frame values within 600 ticks of each other, which makes sense:

~~~~> 327745 (-> v1-9 attack-time) vs 327750 (current-time)
~~~~> 327745 (-> v1-9 attack-time) vs 327755 (current-time)
~~~~> 327745 (-> v1-9 attack-time) vs 327760 (current-time)

But after following the setup that theUsual pointed out, we would see something very strange - the attack-time was in the future, relative to (current-time):

~~~~> 328750 (-> v1-9 attack-time) VS 328527 (current-time)
~~~~> 328750 (-> v1-9 attack-time) VS 328532 (current-time)
~~~~> 328750 (-> v1-9 attack-time) VS 328537 (current-time)

Well, this fits with the extra long cooldowns, but why is it happening?

Clocks​

The game actually keeps track of several different clocks simultaneously, and on a given frame it may or may not tick each clock forward, and by varying rates, depending on the state of the clock and state of the game.

(deftype display (basic)
   ...
   (clock                 clock          22)
   (session-clock         clock          :overlay-at (-> clock 0))
   (game-clock            clock          :overlay-at (-> clock 1))
   (base-clock            clock          :overlay-at (-> clock 2))
   (real-clock            clock          :overlay-at (-> clock 3))
   (frame-clock           clock          :overlay-at (-> clock 4))
   (real-frame-clock      clock          :overlay-at (-> clock 5))
   (target-clock          clock          :overlay-at (-> clock 6))
   (entity-clock          clock          :overlay-at (-> clock 7))
   (part-clock            clock          :overlay-at (-> clock 8))
   (bg-clock              clock          :overlay-at (-> clock 9))
   (camera-clock          clock          :overlay-at (-> clock 10))
   (total-game-clock      clock          :overlay-at (-> clock 11))

For example, the target-clock is not ticked ahead whenever the player pauses the game, whereas real-clock will keep ticking like normal while paused. Or while using Light Jak Flash Freeze, the rates for entity-clock will be reduced so that time moves more slowly for actors other than Jak.

So let's go back to the same-attack-invulnerable-timeout logic. We're checking the attack-time we pulled out of the attack-info-old array - where/how does that attack-time get set? It happens as part of this compute-intersect-info method that we call once we've confirmed an attack should count:

(defmethod compute-intersect-info ((this attack-info) (arg0 object) (arg1 process-drawable) (arg2 process) (arg3 touching-shapes-entry))
  ...
  (when (not (logtest? (-> this mask) (attack-mask attack-time))) 
    (set! (-> this attack-time) (-> *display* base-clock frame-counter)) 

We can see attack-time is set from the base-clock, so let's see if what we're comparing against is also based on the same clock.

time-elapsed? is checking (current-time) AKA (-> PP clock frame-counter). PP here is the current process, at the time when this code is executed - in this case it will be target since we're in the target-log-attack behavior which belongs to target. Since target will be spawned from the *target-pool*, it will have the target-clock as its clock:

(defun set-display ((arg0 display))
  ...
  (set! (-> *target-pool* clock) (-> arg0 target-clock))

So we're comparing time-frames from two different clocks, which starts to lose meaning if one of these clocks is/was paused or is/was running at a different rate than the other.

During various "slow down" effects, the target-clock runs at a reduced rate compared to the base-clock. So every restart mission, every Light Jak Freeze, every Dark Jak special attack, etc causes the target-clock to run slower and fall behind the base-clock.

Once this has happened one or more times, any attack-time will be set based on base-clock, with a value seconds ahead of target-clock. Then in the same-attack-invulnerable-timeout logic, we'll compare our process's clock (target-clock) vs the attack-time, and because of the clock drift, the cooldown will end up being seconds longer than intended!

Most annoying of all is that these clocks won't get reset by starting a new game or going back to the main menu, the only option is to reset your console / close and reopen OpenGOAL.

The Fix​

This is another fairly straightforward fix - we just need to use the same clock on both sides of the comparison. We decided to use the base-clock in this case, so we just modified the same-attack-invulnerable-timeout logic a bit:

-(if (not (time-elapsed? (-> v1-9 attack-time) (the-as time-frame (-> *TARGET-bank* same-attack-invulnerable-timeout))))
+;; og:preserve-this fix clock drift bug which can lead to longer invuln timeouts (use base-clock instead of target-clock)
+(if (not (>= (- (-> *display* base-clock frame-counter) (-> v1-9 attack-time)) (the-as time-frame (-> *TARGET-bank* same-attack-invulnerable-timeout))))
  (return #f)

Since this bug was just an annoyance and didn't have any benefits, it was non-controversial to get this merged into the vanilla jak-project. Other bugs however, such as the invulnerability glitch mentioned earlier, or Citadel Skip in Jak 1, won't be a detriment to casual gameplay and do have value in the speedruns, so you probably won't see them getting fixed. Unless you play the Patched mod 😜

References​

• PR #3978

I was made aware of this glitch in OpenGOAL after we added Individual Level and Hub 2/3 100% speedrun options to the speedrunner menu. For example, players running Hub 2 100% would hit the Blue Sage warp gate button, skip the cutscene, and then run off into Precursor Basin. If they reset the run from there, they would spawn back at the end of Fire Canyon and while entering Rock Village the warp gate cutscene would start playing early, because the button was still pressed from the previous run. Similarly, any orbs or flies collected in Rock Village in the previous run would be missing after resetting with this same setup.

entitys and perm data​

To understand how this glitch happens, we need to cover some fundamentals about how the game keeps track of all these things.

All of the things in the game world that you interact with (lets call them "actors") are spawned as different types of process-drawables, and the majority of them are backed by an entity. I like to think of an entity as the "idea" of an actor, and the process-drawable as the physical representation of the actor. Each entity also has a variety of perm data - this data is used for things like remembering a button has been pushed, or that Plant Boss has been killed, or that you've collected 3 out of 4 Orbs from a crate.

The game tracks this perm data in a couple places. The global *game-info* object has a perm-list which functions as a sort of snapshot of all perm data for the current game. Separate from this, the game has up to 2 levels loaded into memory, and each level holds onto a list of entitys along with their perm data. This data is directly reflected in how the entitys are spawned (or not spawned) as process-drawables in-game, with the actor's state being determined by the entity perm data.

Whenever a level is birthed (i.e. was already loaded, and is being made visible), the game will copy any relevant perm data from *game-info* over to the corresponding entity perm data in the level:

(defmethod birth ((this level))
  "Birth a level to make it alive! It must be loaded."
  ...
  (copy-perms-to-level! *game-info* this)

Whenever a level is deactivated (i.e. made invisible, but still loaded in memory), the game will copy all of the entity perm data for that level into the *game-info* perm-list:

(defmethod deactivate ((this level))
  "Kill the level. This won't remove it from memory."
  ...
  (copy-perms-from-level! *game-info* this)

Whenever you save the game, it will first copy perm data from active levels over to *game-info*'s perm-list. Then it will write the perm-list data into the save file:

(defmethod save-game! ((this game-info) (arg0 game-save) (arg1 string))
  "Update the game-save to have the info from the current game state"
  ...
  (dotimes (s3-0 (-> *level* length))
    (let ((a1-1 (-> *level* level s3-0))) (if (= (-> a1-1 status) 'active) (copy-perms-from-level! this a1-1))))
  ...
  (let ((v1-60 (&+ v1-59 32))
        (s4-2 (-> (the-as (pointer int32) (-> this perm-list)) 0)))
    (let ((a0-39 (the-as game-save-tag (&+ v1-60 0))))
      (set! (-> a0-39 elt-type) (game-save-elt perm-list))
      (set! (-> a0-39 elt-count) s4-2)
      (set! (-> a0-39 elt-size) (the-as uint 16)))
    (let ((s3-4 (&+ v1-60 16)))
      (dotimes (s2-2 s4-2)
        (mem-copy! (the-as pointer (the-as game-save-tag (&+ s3-4 (* s2-2 16))))
                    (the-as pointer (-> this perm-list data s2-2))
                    16))

Whenever you start a new game or load a save file, first the initialize! method will run and make sure to reset *game-info*'s perm-list to length zero, effectively clearing it out:

(defmethod initialize! ((this game-info) (cause symbol) (save-to-load game-save) (continue-point-override string))
  ...
  (set! (-> this perm-list length) 0)

Then if loading a file, it will read the perm data from the file and load it into *game-info*'s perm-list:

(defmethod load-game! ((this game-info) (save game-save))
  "Copy save data from a game-save to a game-info"
  ...
  (let ((data (the-as game-save-tag (-> save tag))))
    (while (< (the-as int data) (the-as int (&-> save tag 0 user-int8 (-> save length))))
      (case (-> data elt-type)
        ...
        (((game-save-elt perm-list))
         (let ((s3-2 (min (-> data elt-count) (-> this perm-list allocated-length))))
           (set! (-> this perm-list length) s3-2)
           (dotimes (s2-0 s3-2)
             (mem-copy! (the-as pointer (-> this perm-list data s2-0)) (&+ (&+ (the-as pointer data) 16) (* s2-0 16)) 16))))

(And then any levels being birthed will get perm-list data from *game-info* copied into their entity perm data)

So what's wrong?​

At first glance it seems like the data is passed between levels ↔️ *game-info* ↔️ game-save in a sensible way, so what's the issue?

Well, let's say you collect an Orb money-2844 in village2, and then you walk into Precursor Basin such that village2 is deactivated but still loaded in memory. If you load a save, or reset a speedrun to a checkpoint that has village2 loaded, what happens?

• village2 is still currently loaded, and has perm data for money-2844 tracking that you collected it
• initialize! runs and clears out *game-info*'s perm-list
• if loading a save, *game-info*'s perm-list is populated from the save file (let's assume the save file has no perm data about money-2844)
• all relevant levels are loaded and birthed (village2 becomes visible)
  • as part of this, *game-info*'s perm-list is copied over to village2's entity perm data
  • BUT *game-info* has no perm data on money-2844, so whatever perm data village2 had is left untouched!

There's an assumption baked in somewhere here that either a level being birthed will have no meaningful perm data, or that it can always be fully reset by copying over perm data from *game-info*. Either way you look at it, the end result is that you can end up with incorrect stale perm data in your level. So Orbs remain collected, buttons remain pushed - you can even cheat at Yakows by putting all but 1 in the pen, loading a save to trigger the glitch, and then you just need to put the 1 remaining Yakow in the pen to get the Cell reward.

The Fix​

So how do we fix the bug? This one's actually pretty simple - since perm data is written to *game-info* when a level is deactivated, and copied back into the level when it's birthed, we don't really need to hold onto any perm data in the level once it's been deactivated. This is true regardless of whether we birth the level by walking back over towards the level, loading a save, or resetting a speedrun.

Lucky for us, there is a debug function reset-actors that does pretty much exactly what we want - it clears out any modified perm data from the active levels. We can copy some of its code into the deactivate function, after we've copied the perm data over to *game-info*:

(defmethod deactivate ((this level))
  "Kill the level. This won't remove it from memory."
  ...
  (copy-perms-from-level! *game-info* this)
  ;; og:preserve-this fully clear entity perm status in the level itself (based on reset-actors)
  ;;  it should be copied back out of game-info on birth to prevent "NG+ glitch"
  (let ((lev-ents (-> this entity)))
    (dotimes (idx (-> lev-ents length))
      (let ((ent (-> lev-ents data idx entity)))
        (update-perm! (-> ent extra perm) 'game (the-as entity-perm-status 1919)))))

And voila, no more NG+ Glitch!

References​

• PR #3690

I did some brief research about Lisp programming syntax (I had one college course where we wrote some Scheme, so I was loosely familiar), and then started reading through Pull Requests on the jak-project GitHub, trying to find smaller code changes that I could follow along with. Eventually I found a GitHub issue listing off various ideas and improvements for Jak 1, and looked for something that sounded approachable, something I could attempt to complete and use to get my feet wet with OpenGOAL.

There was one bullet point that caught my attention as a possible candidate:

Camera invert toggle (for the weirdos).

Understanding controller inputs​

Back in 2021/2022, I hadn't played Jak much in recent years, so I remember the camera controls tripping me up when I started playing regularly through OpenGOAL. Adding settings to toggle inverted camera seemed like something I might want for myself, so I decided I would try to add support for it (for the record, I have since learned the errors of my ways, and now play with default camera controls).

I browsed through all kinds of code, rather blindly at first, trying to understand how controller inputs were read. I saw cpad-pressed? and cpad-hold? throughout the code, and digging deeper I found where the game was reading individual button values, like (-> *cpad-list* 0 square). After awhile it clicked that leftx, lefty, rightx, and righty referred to the X and Y values for the two analog sticks (in particular, I kept seeing rightx/righty referenced in camera code).

99% of the time, these analog stick values were being evaluated using the analog-input function, for example:

(analog-input (the-as int (-> *cpad-list* cpads 0 leftx)) 128.0 48.0 110.0 -1.0)

With some experimentation I figured out that multiplying the result of analog-input by -1 would invert the direction the camera moved.

Extending PC settings​

In the PC port we've added a variety of settings that didn't exist in the original game - things like display resolution, aspect ratio, MSAA, cutscene skips, etc. Naturally, this is where it made sense to add settings to toggle inverted camera controls as well. Typically boolean (on/off) settings like this are represented as symbols (either #t or #f).

Now the Jak games are a bit special, in that camera controls are already inverted compared to the defaults in most modern games - except for first person horizontal controls! At first I had only 2 settings, for vertical and horizontal camera inversion, but later I extended this to 4 settings so that players could configure first and third person controls independently:

(deftype pc-settings (basic)
   ...
   (first-camera-h-inverted?    symbol) ;; first-person horizontal camera inverted
   (first-camera-v-inverted?    symbol) ;; first-person vertical camera inverted
   (third-camera-h-inverted?    symbol) ;; third-person horizontal camera inverted
   (third-camera-v-inverted?    symbol) ;; third-person vertical camera inverted

There are a few other bits of code to update when introducing new settings (resetting settings, reading from file, writing to file, etc), but they're not very interesting so I'll skip over them here. You can check my PRs below or search the codebase for one of the setting field names (e.g. first-camera-h-inverted?) if you're curious.

The other bit of work relating to adding new settings is hooking them up to options in the pause menu, known internally as the progress menu. I've written about this process before in the Checkpoint Randomizer modding example, so I won't rehash it here.

Modifying vanilla code​

Since the decompiler is always evolving and improving, occasionally we will re-decompile all or some of the GOAL source code. So whenever we make edits to OpenGOAL code, we try to edit the vanilla code as little as possible, or leave special comments as markers indicating that the code was changed in the PC port. This way if the code is decompiled again, our edits are either unaffected, or we can detect that markers were removed and go back and restore the PC port edits. For example, we adjusted some HUD sprite positioning for wider aspect ratios:

;; og:preserve-this Positioning of first person hud sprites
(when (or (-> *pc-settings* use-vis?) (= s5-0 2))
    (spawn (-> this particles s5-0 part) *null-vector*)))

Originally I added some code directly in the analog-input function to check the PC settings and decide whether to multiply by -1 and invert the controls, but the downside to this approach was that by the time we're in analog-input, we no longer have context as to whether we're evaluating the inputs for left/right or for up/down. Since we now have 4 separate PC settings for camera inversion, I would need to make some changes upstream of the analog-input function itself to support this properly.

The solution I landed on was to create 4 macros that wrapped the analog-input call:

(defmacro analog-input-horizontal-first (in offset center-val max-val out-range)
  "Same as analog-input but respects First-Person Horizontal camera control setting."
  `(#if PC_PORT ;; first-person horizontal is NOT inverted in original game
    (* (if (-> *pc-settings* first-camera-h-inverted?) -1.0 1.0) (analog-input ,in ,offset ,center-val ,max-val ,out-range))
    (analog-input ,in ,offset ,center-val ,max-val ,out-range)))

(defmacro analog-input-vertical-first (in offset center-val max-val out-range)
  "Same as analog-input but respects First-Person Vertical camera control setting."
  `(#if PC_PORT ;; first-person vertical is already inverted in original game
    (* (if (-> *pc-settings* first-camera-v-inverted?) 1.0 -1.0) (analog-input ,in ,offset ,center-val ,max-val ,out-range))
    (analog-input ,in ,offset ,center-val ,max-val ,out-range)))

(defmacro analog-input-horizontal-third (in offset center-val max-val out-range)
  "Same as analog-input but respects Third-Person Horizontal camera control setting."
  `(#if PC_PORT ;; third-person horizontal is already inverted in original game
    (* (if (-> *pc-settings* third-camera-h-inverted?) 1.0 -1.0) (analog-input ,in ,offset ,center-val ,max-val ,out-range))
    (analog-input ,in ,offset ,center-val ,max-val ,out-range)))

(defmacro analog-input-vertical-third (in offset center-val max-val out-range)
  "Same as analog-input but respects Third-Person Vertical camera control setting."
  `(#if PC_PORT ;; third-person vertical is already inverted in original game
    (* (if (-> *pc-settings* third-camera-v-inverted?) 1.0 -1.0) (analog-input ,in ,offset ,center-val ,max-val ,out-range))
    (analog-input ,in ,offset ,center-val ,max-val ,out-range)))

From here, I just needed to replace the various camera-related analog-input calls throughout the codebase with the appropriate macro, based on the particular type of camera (first vs third person) and which controller input was being read (rightx vs righty, i.e. horizontal vs vertical).

For example, cam-eye which is the first person camera used when you press triangle to use your goggles:

;; main first-person camera
(defstate cam-eye (camera-slave)
  ...
  :code
    (behavior ()
      ...
      (let ((f30-0 (analog-input-horizontal-first ;; og:preserve-this changed for pc port
                      (the-as int (+ (-> *cpad-list* cpads 0 rightx) -256 (-> *cpad-list* cpads 0 leftx)))
                      (the-as float 0.0)
                      (the-as float 48.0)
                      (the-as float 110.0)
                      (the-as float -1.0)))
            (f0-0 (analog-input-vertical-first ;; og:preserve-this changed for pc port
                    (the-as int (+ (-> *cpad-list* cpads 0 righty) -256 (-> *cpad-list* cpads 0 lefty)))
                    (the-as float 0.0)
                    (the-as float 48.0)
                    (the-as float 110.0)
                    (the-as float -1.0))))

And a more unique example, cam-string-joystick, the main third person camera used during 99% of gameplay:

(defun cam-dist-analog-input ((arg0 int) (arg1 float))
  (let ((f0-0 0.0))
    (cond
      ((< arg0 28) (set! f0-0 (- (fmin arg1 (* 0.083333336 (- 28.0 (the float arg0)) arg1)))))
      ((< 160 arg0) (set! f0-0 (fmin arg1 (* 0.0125 (+ -160.0 (the float arg0)) arg1)))))
    ;; changed for pc port, only used for Third-Person Vertical camera
    (#if PC_PORT (* (if (-> *pc-settings* third-camera-v-inverted?) 1.0 -1.0) f0-0) f0-0)))

;; main third-person camera
(defbehavior cam-string-joystick camera-slave ()
  (logand! (-> self options) -257)
  (let ((f28-0 (cam-dist-analog-input (the-as int (-> *cpad-list* cpads 0 righty)) (the-as float 0.05)))
    ...
    (let ((f0-29 (analog-input-horizontal-third ;; changed for pc port
                   (the-as int (-> *cpad-list* cpads 0 rightx))
                   (the-as float 128.0)
                   (the-as float 32.0)
                   (the-as float 110.0)
                   (* 21845.334 (seconds-per-frame))))

The original code for cam-string-joystick uses cam-dist-analog-input in place of the regular analog-input for vertical camera controls, but since this is the only place cam-dist-analog-input is used, we can get away with only checking our third-camera-v-inverted? setting here.

There were a few additional camera functions where I swapped out analog-input, but the changes were quite similar to the above. These included things like cam-free-floating for the debug freecam, and cam-periscope used for the FJ mirrors and Misty cannon.

Contributing to jak-project​

I want to briefly call out that I didn't get all these changes right on the first try 😅. I recall nervously opening a PR with my initial changes, and waiting around to see what would happen next.

The next day I got some good feedback from a couple of the main developers, water111 and dass. They gave me a quick breakdown of some compiler behavior I was misunderstanding, and we had some back and forth about how to improve my changes. A couple commits later and my changes were merged in!

I circled back with another cleanup PR a week or so later after talking more with dass. We changed some language in the pause menu (Inverted vs Flipped), added a "Restore Default" option, and added some sound effects.

I bring this all up because I really think the best way to get started is just to dive in and try something, even if you make mistakes along the way. The developers/community are really supportive and want to teach OpenGOAL to others who demonstrate that they are passionate about it.

Looking back​

When someone in the Discord asks how best to get started with OpenGOAL development/modding, I often bring this up as an example - it was small in scope, but still gave me an opportunity to explore the codebase. I got to see how controller inputs are processed and used for in-game logic, I got to work with PC settings and the pause menu, and (maybe most importantly) it forced me to just generally get familiar with the OpenGOAL development loop.

I still consider this my first OpenGOAL mod, even though the changes ended up in vanilla jak-project. Getting to write some new code, and then seeing my changes work in-game, it was a really exciting moment for me. Shortly after this I found out about Mike & Zed's Checkpoint Randomizer, and was able to connect with them and start contributing to the mod. To this day, that feeling of seeing my code changes reflected in-game is what keeps me interested in working on OpenGOAL.

I actually got to port these camera inversion changes over to Jak 3 recently, which was a nice little full-circle moment 😊

References​

• PR #1355 (main changes)
• PR #1361 (clean up)
• PR #4171 (Jak 3 changes)

As this is a pretty big release with a lot of moving parts, I figured I’d write a little blog to answer some of the questions you might have or get some behind the scenes if you’re interested, as I’ve mostly been working in the dark. Only a select few have seen different steps in the progress of this project.

FAQ​

I’ll start with a small Q&A of probably the most likely questions:

How long have you been working on this?​

I started working on this mod in June 2022, with the first release being published on January 9th 2023.

After that first release, I gave myself a few weeks off the project before diving back into it.

There was around 350 hours of work done on the mod for the first release. Steam has actually stopped tracking my hours on Blender a while back, so I don’t have a precise estimate, but I’ve been working on this new release for over 10 months at a much higher density than the first release so I think I can confidently say that the total time I’ve been working on this for is easily over 1000 hours, probably in the ballpark of 1300 to 1600 hours total.

Are you working on this on your own?​

Not completely nowadays, see the “New Contributors” section for more info on this.

How is this made?​

I’ll try to be very concise here as this topic could be several pages in itself. In short, you need to create the level’s geometry using 3D modeling software. I’m using Blender myself, as the OpenGOAL devs made some tools specifically for Blender. And also because Blender is an amazing software, don’t listen to Zed.

Along with the level geometry, you’ll also need to edit some code in OpenGOAL to be able to add your level in but also to add collectables, water, sounds, particles and everything else.

Most of the geometry I’ve been using for my level was custom-made by me. The textures are a mix of edited textures from the main game and fully custom ones as well.

I’ve made an incomplete and now somewhat outdated tutorial which should give you a more complete idea.

Are there any limitations?​

At the moment, some custom level tools are not available yet and are the reasons I haven’t implemented some specific aspects into the game. Hopefully, this release motivates some of the devs to work on these things. wink wink

Here’s a short list of the main missing features at the moment:

• Importing custom models and skeleton to replace/create new actors.
• Texture transparency for things like grass, ice and others.
• Reflections for things like Precursor Metal.
• Sky rendering in custom levels. (Need to use a “fake” sky sphere instead for the time being.)
• Nav meshes, Jak 1’s system to make enemies or NPCs move around an area, avoid obstacles and path toward Jak. This means most enemies can’t be added at the moment. A few enemies don’t use nav meshes and can be implemented.

Are you looking for help for anything?​

I don’t have anything in particular right now where I do need help. If you do have a specific skill set that you think would be useful for the project, then you can always contact me, but be aware that it’s pretty likely to be a negative answer. You can read the “New Contributors” section for more info on my mindset in this project overall.

Can we support you or the project with donations?​

Short answer, no.

Long answer; this is a passion project for me, while I’ve put several hundreds of hours into this and will most likely put in a lot more, I do not see it as wasted time because I’m not making money out of it.

I’ve always loved making levels for games. (CS 1.6, LBP, Portal 2, Dustforce and I’m probably forgetting some) So even just this whole thing being possible thanks to OpenGOAL is a dream come true for me. Adding money to the mix would change how I view the project and it’d definitely lose something. I’m making money elsewhere. I'd rather this passion project stay exactly that.

New Contributors​

I’ll start this part with some history on how this project started. As said in the FAQ, this project was started pretty much as soon as custom level became a possibility in OpenGOAL. I even started writing and drawing some ideas a bit before, while waiting for the feature.

My plans from the beginning have been stupidly large in terms of scope and… they’ve not really changed much. If anything, they’ve grown a bit over time.

I was, from the very start, fully committed to pretty much doing everything on my own. Which sounds kind of ridiculous when compared to the scale of the project, but the rate at which I’ve been able to work and complete things, this has actually been pretty good. I don’t think I could have expected to have done this much after 17 months.

The main reason for my mindset is pretty much an egotistical one. This is probably going to be one of my biggest, if not the biggest, projects I will ever make. I want this to be almost entirely my artistic vision and to have control over every minute detail. Having people help, even for work that seems to be repetitive and unimportant, felt like I’d lose some control over the creative process and, as silly as that might sound, was not something I was ready to accept.

When I released the first version with Crystal Cave, I got a lot of people messaging me, offering help. There was one aspect where I was actually willing to get some help, as it’s an aspect I don’t think I’d be able to do myself: Music.

Pex​

One of the people messaging me about offering help was none other than Pex. Maybe the name doesn’t ring a bell, but you’ve probably heard some of the soundtracks he’s made for the cut Jak and Daxter levels mentioned in the Design Bible. You can listen to them here.

I don’t think I could have hoped for someone better for the job, short of the original composer of Jak 1’s OST. Pex has done an incredible job on the soundtrack for TFL so far and been great to work with.

Hat Kid​

Another aspect of the project I was having trouble with was the programming aspect. While I have some very small basic knowledge on it, it was a huge undertaking to learn everything I needed to do some of the things I would like to be in the mod.

However, about 3 months ago. I decided to message Hat Kid to see if he wanted to be more involved in the project to help me achieve all those crazy things I wanted to implement.

I can’t thank him enough for all the work he’s been able to do in such a short period of time, which includes:

• Adding support for ambients in custom levels. Those allow us to add things like ambient audio, but also level name pop-ups.
• Fixing many crashes related to custom levels.
• Adding support to include models and textures in custom levels. This allows us to use almost every single actor in the game in custom levels. Probably the most significant change for custom levels over the whole year.
• Creation of the custom “Super Eco” powers.
• Custom music system roughly based on Barg and Zed’s audio hack with automatic level detection (only need to add the level name to a list and the corresponding audio file), pausing, volume adjustment based on the in-game music volume setting and fade in/fade out.
• Custom level hint system, also using the same (heavily modified) audio hack that is very customisable (can repeat voice lines, interrupt other playing ones if wanted, only play based on special conditions/task status/etc.).
• Sinking and emerging particle spawner for platforms.
• And finally, a working gondola!

I’ve probably even missed some other things, but as you can see, this list is getting really big. You can find a much more detailed explanation in Hat Kid’s written section later in this blog.

He’s also helped me a lot with some of the custom code I tried to write, such as the electric arcs in Crystal Cave.

Voice Actors​

Maybe a bit of a controversial topic, but I’ll let you know how I got there to clear things up.

I’ve wanted to add voice lines to the project, both to tell the story I’ve been writing for this mod as well as to help the player, so they’re not completely lost and have a general idea of what they need to do.

To do so, I’ve actually had a few different ways to approach the situation. I first wanted to actually message the original voice actors of Jak 1. First, to see if they would be interested, but then, more realistically, if they were fine with me using AI voices trained on their work.

I was not really able to find contact information for Daxter’s and Keira’s voice actors and Samos’ VA, Warren Burton, is not with us anymore.

Which made me decide to look for impersonators instead. I’ve sent out a form on a few different platforms but got very few replies. One of the few I got was from a guy that has already been working on AI models for most of the characters in the franchise. At that point, it was one of my last options if I did want to have voice lines, so I’ve decided to go that route.

As mentioned earlier, I’m not going to make any money out of this project, so I’m not trying to profit off of these voice actors’ work, but if any of them wish for me to remove the lines from the mod, it’ll be done the next day.

The models have been trained by Netguard and here’s the voice actor for each of the characters, as you do still need someone acting out the lines for the best results:

• Michael Gross as Daxter
• Nohmyy as Keira
• Liam Donovan and Michael Gross as Samos

Crystal Cave Improvements​

Crystal Cave has received a few additions and finishing touches.

Paths!​

Custom paths have been figured out by Luminar Light a couple of months ago. Thanks to them, I’ve been able to make good use of this feature. The platforms in Crystal Cave are now moving!

Functional Water​

Another big missing part of Crystal Cave has now also been added, water will now work thanks to a change Hat Kid also made not too long after the first release. The visual for those is not proper yet as right now, I’d need to be able to match the area perfectly with some water that already exists in the game as those are just 3d models and we can’t import custom ones yet as mentioned earlier.

Custom Jak Lighting​

As some have noticed, some entities in Crystal Cave have custom lighting depending on the different areas of the level that have different lighting. This has also been added to Jak as he moves around the level. It’s a pretty small detail, but works really well, so Jak doesn’t look out of place.

Super Green Eco and New Cell​

The first Super Green Eco you’ll stumble upon. It makes you invincible to most damage sources and temporarily increases the maximum Green Eco pill count to 100, even after the powerup runs out, allowing you to take five hits. It allows you to collect the extra cell without taking damage on the Dark Eco infected ground.

Electric Arc in the Precursor Ship​

Mostly heavily edited code from the flame pots seen in the robot room of Spider Cave with custom particle work, adding a new hazard in the ship area.

Ambients​

Ambients allow us to display the name of the level as well as playing some ambient sounds all over the level. The waterfalls aren’t silent anymore!

New Levels!​

I’m sure most people were expecting a new level in this update, but hopefully the second one was a nice surprise as well!

These new levels are obviously the bulk of the work of the past 10 months.

Crescent Top​

When you come out of Crystal Cave, you will arrive on this crescent-shaped plateau, hence the name.

It serves as a hub to other levels. However, as opposed to the base game’s hubs that have pretty limited gameplay, I’ve wanted to make it a fully fleshed out level with loads of different gameplay features.

The level is built to be a mix of on-foot and zoomer gameplay within the same area. This has been quite a challenging idea, as a fun area to traverse on the zoomer isn’t as good on foot and vice versa. I’ve tried to make the level flow really well when you do know the “route”, so I hope it won’t feel too sluggish when playing through it for the first time.

Here’s a timelapse of a few recordings I took while working on Crescent Top played at 50 times the speed.

Energy Bay​

The second new level where you’ll have to go before being able to finish Crescent Top itself, as you’ll unlock the zoomer there.

A much smaller level than Crescent Top. Probably took me less than half of the time that Crescent Top took me.

It’s the home of the second Super Eco power you’ll find, Super Red Eco, which has many features, both on the zoomer and on foot that makes it hopefully a lot more useful than normal Red Eco!

Hat Kid’s Section​