March 2024 Recap

Because I’m not able to keep up with writing a post every time I have something interesting to share, I’ll sometimes just briefly summarize highlights over the past 1/2/3/4 weeks; in this case, the month of March.

“csv record too large” error

When attempting to read a file using an open source csv library called Sylvan.Data.Csv I encountered a CsvRecordTooLargeException.The error kept occurring at the same line number: 598,191.

The exception that is thrown when a single CSV record is too large to fit in the read buffer. CsvRecordTooLargeException is typically an indication of a incorrectly formatted CSV file. Most CSV files should easily fit the constraints of the default buffer size. It is possible to increase the buffer size for exceptional cases. source

Something was obviously not formatted correctly. I was unable to read the whole line without it throwing an exception, even with plain PowerShell commands. So, I wrote a .NET Console application script to read line-by-line until the problematic line and then read one character at a time.

static void Main(string[] args)
{
    string filePath = @"path_to_the_csvs\PDP_20220601.csv";
    int lineNumberThatThrows = 598191;

    string nullCharCount = GetNullCharCount(filePath, lineNumberThatThrows);
    Console.WriteLine($"Null char count: {nullCharCount}");
}

static string GetNullCharCount(string filePath, int lineNumberThatThrows)
{
    using (StreamReader reader = new StreamReader(filePath))
    {
        int currentLineNumber = 1;

        // Read lines normally until reaching the desired line number
        while (currentLineNumber < lineNumberThatThrows)
        {
            reader.ReadLine();
            currentLineNumber++;
        }

        // Read characters one by one to see what's going on...
        long nullCharCount = 0;
        int validCharCount = 0;
        int nextChar;
        while (true)
        {
            if ((nextChar = reader.Read()) == -1)
            {
                Console.WriteLine("End of file reached.");
                break;
            }
            char character = (char)nextChar;
            if (character == '\0')
            {
                nullCharCount++;
                continue;
            }
            validCharCount++;
        }

        Console.WriteLine($"Valid char count: {validCharCount}");

        return nullCharCount.ToString();
    }
}

The output was

End of file reached.
Valid char count: 240
Null char count: 11_464_439_842

That’s over 11 billion null characters 🤯 With the diagnosis resolved, we reported the corrupted file and moved on.

IsPointerOverGameObject-related bug

After removing the new Input System package from a project, IsPointerOverGameObject() started always returning false on iOS. This caused bug where the camera would pan around when the user is dragging UI (uGUI), like a slider or scroll view. Turns out there’s an overload where you can supply a “pointerId”, so I just needed to supply the fingerId:

EventSystem.current.IsPointerOverGameObject(Input.GetTouch(0).fingerId)

I guess the new Input System handles that behind the scenes. But you may be wondering why we removed it, in the first place. I’m a fan of the Input System package so I added it when I joined the project. But we just never got around to migrating existing code to use it, so I decided to drop it to prevent unnecessary work being done in the background.

Getting rid of per-frame allocations

As my team and I are getting close to wrapping up the second phase of a project, I decided to go “allocation hunting” in the profiler to seek out optimization opportunities. The easiest ones to spot are those that occur every frame, and I found 3 this time, all related to delegates.

The first one was related to sorting popups by distance from the camera.

private void Update()
{
    ...
    _popups.Sort((a, b) => { });
    ...
}

The problem here is that a lambda gets allocated every frame. It’s using the Sort(Comparison<T>) overload, where Comparison<T> is a custom delegate. I’ve resolved issues like this before by caching the delegate, but I noticed there’s also an IComparer<T> overload, so out of curiosity I decided to give that a shot first.

PopupSortComparer _sortComparer;

private void Update()
{
    ...
    _popups.Sort(_sortComparer);
    ...
}

private class PopupSortComparer : IComparer<PopupBase> { ... }

where _sortComparer is initialized in Awake.

But profiling revealed that even supplying a cached comparer to List<T>.Sort results in per-frame-allocations. After turning on deep profiling, I found that the source is ArraySortHelper<T>.Sort.

So I checked the .NET source code and found that it’s due to the method group: comparer.Compare being passed to IntrospectiveSort.

ArraySortHelper<T>.IntrospectiveSort(keys, comparer.Compare);

Caching the delegate gets rid of repeated allocations, as expected, but I’m glad I tried out IComparer for the sake of awareness.

Comparison<PopupBase> _sortComparison;

private void Update()
{
    ...
    _popups.Sort(_sortComparison);
    ...
}

Graph and Chart Allocations

I found that the other 2 sources of per-frame-allocations were the same kind as the first: delegates. Except this time happening in an Asset Store library called Graph and Chart - Lite Edition (which is a great library, by the way; it saved us a lot of time).

bool DoSlider(Slider s)
{
    return s.UpdateSlider(this);
}

// Called by Unity's Update method
protected void UpdateSliders()
{
    mSliders.RemoveAll(DoSlider);
}

See the problem? The method group DoSlider is passed to RemoveAll every frame.

The fixed version only allocates once at init time:

private Predicate<Slider> mPredicate;

private void Awake()
{
    mPredicate = DoSlider;
}

bool DoSlider(Slider s)
{
    return s.UpdateSlider(this);
}

protected void UpdateSliders()
{
    mSliders.RemoveAll(mPredicate);
}

Plastic SCM commit messages to release notes

My team and I are using Plastic SCM for our current project, which is my first experience with it. And we’re at the phase where we’re creating regular releases. But the release notes were manual. I’ve created many GitHub Workflows and Actions throughout my career, including conventional commit parsers for generating release notes, but unfortunately nothing I could use for Plastic. I eventually decided to take a day to learn the CLI commands and try to piece together a minimal solution. I have a little experience using Python, so that was my language of choice. I didn’t want to spend much time on this, so I tried using Chat GPT, but it really sucks at the Plastic CLI syntax. I finally got something decent working. Not pretty but gets the job done.

Here’s the gist for those that are interested.

The basic logic is

changesetid = get_last_label()
if changesetid:
    commit_messages = get_commit_messages_since_label(changesetid)
    commit_messages = filter_commit_messages(commit_messages)
    message_groups = group_commit_messages(commit_messages)
    print("Commit messages since last label:")
    for prefix, messages in message_groups.items():
        print(f"{prefix}")
        for message in messages:
            print(message)
else:
    print("No labels found.")

Google Play Developer Account Warning

Back in January I received a warning notification about my Google Play Developer account, saying it’s at risk of being closed because I haven’t published or updated any apps in a long time. Apparently, they started cracking down on dormant accounts a few years ago.

The deadline for me to publish or update an app was set at March 23rd. I had one Korean study app published sometime between 2016 and 2018, but I eventually unpublished it because of changing interests and limited time, and haven’t done anything else with my account since. It was developed with Android Studio + Java. Every once in a while, within the past five years, I experiment with the idea of doing a remake with .NET Maui (the successor to Xamarin). Anyways, I discovered that even if my account gets closed, I’m allowed to sign up for a new one with a different email. Even so, I decided to try to beat the deadline by publishing a minimal Korean verb conjugator app, with the intention of later evolving it into that remake I mentioned. This included learning how to write a privacy policy because it’s a requirement these days. The first three months of this year have been crazy because of work, so I actually did most of the development in the last couple weeks and finally submitted an apk for publishing on March 22nd 😄 It’s still in review, but looks like submission was enough to save my account because I found the following message in my notification archive:

Maybe this was the nudge I needed to get back into creating/publishing projects. After this Korean app, I’ll have my sights set on developing games.