“csv record too large” error

When attempting to read a file using an open source csv library called Sylvan.Data.Csv I encountered a CsvRecordTooLargeException.The error kept occurring at the same line number: 598,191.

The exception that is thrown when a single CSV record is too large to fit in the read buffer. CsvRecordTooLargeException is typically an indication of a incorrectly formatted CSV file. Most CSV files should easily fit the constraints of the default buffer size. It is possible to increase the buffer size for exceptional cases. source

Something was obviously not formatted correctly. I was unable to read the whole line without it throwing an exception, even with plain PowerShell commands. So, I wrote a .NET Console application script to read line-by-line until the problematic line and then read one character at a time.

static void Main(string[] args)
{
    string filePath = @"path_to_the_csvs\PDP_20220601.csv";
    int lineNumberThatThrows = 598191;

    string nullCharCount = GetNullCharCount(filePath, lineNumberThatThrows);
    Console.WriteLine($"Null char count: {nullCharCount}");
}

static string GetNullCharCount(string filePath, int lineNumberThatThrows)
{
    using (StreamReader reader = new StreamReader(filePath))
    {
        int currentLineNumber = 1;

        // Read lines normally until reaching the desired line number
        while (currentLineNumber < lineNumberThatThrows)
        {
            reader.ReadLine();
            currentLineNumber++;
        }

        // Read characters one by one to see what's going on...
        long nullCharCount = 0;
        int validCharCount = 0;
        int nextChar;
        while (true)
        {
            if ((nextChar = reader.Read()) == -1)
            {
                Console.WriteLine("End of file reached.");
                break;
            }
            char character = (char)nextChar;
            if (character == '\0')
            {
                nullCharCount++;
                continue;
            }
            validCharCount++;
        }

        Console.WriteLine($"Valid char count: {validCharCount}");

        return nullCharCount.ToString();
    }
}

The output was

End of file reached.
Valid char count: 240
Null char count: 11_464_439_842

That’s over 11 billion null characters 🤯 With the diagnosis resolved, we reported the corrupted file and moved on.

IsPointerOverGameObject-related bug

After removing the new Input System package from a project, IsPointerOverGameObject() started always returning false on iOS. This caused bug where the camera would pan around when the user is dragging UI (uGUI), like a slider or scroll view. Turns out there’s an overload where you can supply a “pointerId”, so I just needed to supply the fingerId:

EventSystem.current.IsPointerOverGameObject(Input.GetTouch(0).fingerId)

I guess the new Input System handles that behind the scenes. But you may be wondering why we removed it, in the first place. I’m a fan of the Input System package so I added it when I joined the project. But we just never got around to migrating existing code to use it, so I decided to drop it to prevent unnecessary work being done in the background.

Getting rid of per-frame allocations

As my team and I are getting close to wrapping up the second phase of a project, I decided to go “allocation hunting” in the profiler to seek out optimization opportunities. The easiest ones to spot are those that occur every frame, and I found 3 this time, all related to delegates.

The first one was related to sorting popups by distance from the camera.

private void Update()
{
    ...
    _popups.Sort((a, b) => { });
    ...
}

The problem here is that a lambda gets allocated every frame. It’s using the Sort(Comparison<T>) overload, where Comparison<T> is a custom delegate. I’ve resolved issues like this before by caching the delegate, but I noticed there’s also an IComparer<T> overload, so out of curiosity I decided to give that a shot first.

PopupSortComparer _sortComparer;

private void Update()
{
    ...
    _popups.Sort(_sortComparer);
    ...
}

private class PopupSortComparer : IComparer<PopupBase> { ... }

where _sortComparer is initialized in Awake.

But profiling revealed that even supplying a cached comparer to List<T>.Sort results in per-frame-allocations. After turning on deep profiling, I found that the source is ArraySortHelper<T>.Sort.

So I checked the .NET source code and found that it’s due to the method group: comparer.Compare being passed to IntrospectiveSort.

ArraySortHelper<T>.IntrospectiveSort(keys, comparer.Compare);

Caching the delegate gets rid of repeated allocations, as expected, but I’m glad I tried out IComparer for the sake of awareness.

Comparison<PopupBase> _sortComparison;

private void Update()
{
    ...
    _popups.Sort(_sortComparison);
    ...
}

Graph and Chart Allocations

I found that the other 2 sources of per-frame-allocations were the same kind as the first: delegates. Except this time happening in an Asset Store library called Graph and Chart - Lite Edition (which is a great library, by the way; it saved us a lot of time).

bool DoSlider(Slider s)
{
    return s.UpdateSlider(this);
}

// Called by Unity's Update method
protected void UpdateSliders()
{
    mSliders.RemoveAll(DoSlider);
}

See the problem? The method group DoSlider is passed to RemoveAll every frame.

The fixed version only allocates once at init time:

private Predicate<Slider> mPredicate;

private void Awake()
{
    mPredicate = DoSlider;
}

bool DoSlider(Slider s)
{
    return s.UpdateSlider(this);
}

protected void UpdateSliders()
{
    mSliders.RemoveAll(mPredicate);
}

Plastic SCM commit messages to release notes

My team and I are using Plastic SCM for our current project, which is my first experience with it. And we’re at the phase where we’re creating regular releases. But the release notes were manual. I’ve created many GitHub Workflows and Actions throughout my career, including conventional commit parsers for generating release notes, but unfortunately nothing I could use for Plastic. I eventually decided to take a day to learn the CLI commands and try to piece together a minimal solution. I have a little experience using Python, so that was my language of choice. I didn’t want to spend much time on this, so I tried using Chat GPT, but it really sucks at the Plastic CLI syntax. I finally got something decent working. Not pretty but gets the job done.

Here’s the gist for those that are interested.

The basic logic is

changesetid = get_last_label()
if changesetid:
    commit_messages = get_commit_messages_since_label(changesetid)
    commit_messages = filter_commit_messages(commit_messages)
    message_groups = group_commit_messages(commit_messages)
    print("Commit messages since last label:")
    for prefix, messages in message_groups.items():
        print(f"{prefix}")
        for message in messages:
            print(message)
else:
    print("No labels found.")

Google Play Developer Account Warning

Back in January I received a warning notification about my Google Play Developer account, saying it’s at risk of being closed because I haven’t published or updated any apps in a long time. Apparently, they started cracking down on dormant accounts a few years ago.

The deadline for me to publish or update an app was set at March 23rd. I had one Korean study app published sometime between 2016 and 2018, but I eventually unpublished it because of changing interests and limited time, and haven’t done anything else with my account since. It was developed with Android Studio + Java. Every once in a while, within the past five years, I experiment with the idea of doing a remake with .NET Maui (the successor to Xamarin). Anyways, I discovered that even if my account gets closed, I’m allowed to sign up for a new one with a different email. Even so, I decided to try to beat the deadline by publishing a minimal Korean verb conjugator app, with the intention of later evolving it into that remake I mentioned. This included learning how to write a privacy policy because it’s a requirement these days. The first three months of this year have been crazy because of work, so I actually did most of the development in the last couple weeks and finally submitted an apk for publishing on March 22nd 😄 It’s still in review, but looks like submission was enough to save my account because I found the following message in my notification archive:

Maybe this was the nudge I needed to get back into creating/publishing projects. After this Korean app, I’ll have my sights set on developing games.

In order to make this as understandable as possible, I’ll be using simplified versions of the actual code, and excluding parts that would otherwise distract us from the main focus.

This code is from a Unity project but it doesn’t touch any Unity-specific API, so you’ll probably see me using Console.WriteLine to demonstrate printing.

So without further ado, here’s the snippet of interest:

1
2
3
4
5
6
7
8
9
10
var data = new TValue[structCount];
// TODO: Find out why ReadArray is returning wrong values but reading one-by-one is fine.
//accessor.ReadArray(position: 0, array: data, offset: 0, count: data.Length);
var structSize = Marshal.SizeOf<T>();
for (var i = 0; i < structCount; i++)
{
    var byteOffset = i * structSize;
    accessor.Read<TValue>(byteOffset, out var value);
    data[i] = value;
}

Line 3 is the statement I had to comment out because it wasn’t working as expected, while the for loop is the “unsatisfying workaround”. Unsatisfying for a couple reasons:

1. reading one-by-one is not as performant
2. not knowing the “why”

accessor is of type MemoryMappedViewAccessor. I’ll cover this in a future post. For now, just know that it’s acting as a binary file reader.

The commented out code works fine when the struct type contains only ints, floats, doubles, etc. The issue only surfaced when it tried to read a struct type that contains boolean values. Let’s walk through an example using the following struct definition.

struct MyStruct
{
    public bool MyBool;
}

We’ll start by writing 2 instances of this struct to a binary file.

public static void WriteStructs<T>(T[] structs)
    where T : unmanaged
{
    var structSize = Marshal.SizeOf<T>();
    var bytes = new byte[structs.Length * structSize];
    // convert structs to bytes...
    File.WriteAllBytes(BinaryFilePath, bytes);
}

var myStructs = new MyStruct[]
{
   new() { MyBool = true },
   new() { MyBool = true }
};
WriteStructs(myStructs);

This results in a binary file with a size of 8 bytes.

What? 8 bytes for 2 bool values?

Although the bool type in C# is only 1 byte in size (sizeof(bool) == 1), the CLR defaults to marshalling it as the unmanaged BOOL type. This is the size you get when you call Marshal.SizeOf. BOOL is a typedef in the Windows SDK headers for an int, which is 4 bytes in size. Why? Because these headers were written for the C language at a time that this language did not have a first-class Boolean type. It does now, but the decisions are fixed in stone for backwards-compatibility reasons. The CLR marshals bool types this way in order to be compatible with Windows API functions that use BOOL values. source

Okay, good to know. Let’s just roll with this for now.

Printing out the bytes gives the following: 1,0,0,0,1,0,0,0

Now it’s clear why the for loop in the first snippet works fine: it uses Marshal.SizeOf<T> just like the WriteStructs method. The 2 bool values we get are True,True when printing the result:

var str = string.Join(",", myStructs.Select(x => x.MyBool));
Console.WriteLine(str);

But what if we try using commented out code?

accessor.ReadArray(position: 0, array: data, offset: 0, count: data.Length);

The 2 bool values end up being True,False 🤔

If we take a look at the ReadArray source code, we can see a call to SafeBuffer.AlignedSizeOf<T>(), which ultimately calls sizeof(T).

sizeof(MyStruct) returns 1. That means ReadArray was only reading the first 2 bytes instead of all 8. Being that the file bytes are 1,0,0,0,1,0,0,0, it now makes sense why it returned True,False. We can try reading 8 structs from that same file to further prove this theory.

accessor.ReadArray(0, data, 0, 8);
// returns: True,False,False,False,True,False,False,False

This brings us to the difference between the two.

sizeof(T) returns the size of a type in managed memory, and Marshal.SizeOf returns the size of a type in unmanaged memory.

sizeof(T) and Unsafe.SizeOf<T> are equivalent in the sense that they both return the managed size of a type, and they even compile down to the same IL instruction. The only difference is that sizeof only works when T is unmanaged, and requires the unsafe keyword when used with custom structs.

Conclusion

So the bug was caused by mixing two different interpretations of size: managed and unmanaged. The binary utility that this code is part of has nothing to do with interop - it only reads and writes C# types - so Marshal.SizeOf is the wrong tool for the job. After swapping out all the Marshal.SizeOf<T> calls for Unsafe.SizeOf<T> I was able to replace the workaround with ReadArray, offically resolving the TODO comment.

var data = new TValue[structCount];
accessor.ReadArray(position: 0, array: data, offset: 0, count: data.Length);

Unravelling this bug would have been a daunting task if I used a struct with many different fields and a binary file with thousands of bytes. But as with any problem, breaking it down simplifies the debugging experience a great deal.

• easily pinpoint when a bug was introduced
• avoid “surprises” when someone requests a build

This includes early in development; not just when getting close to a release. The importance is magnified when building for IL2CPP because of managed code stripping, especially when reflection is in use. I recently did one of these routine checks for a project I’m currently assigned to at work (early in development). It had been 1-2 weeks since my last check, and let’s just say it resulted in enough content for this blog post 😅 There were three main issues that came up.

Awake vs OnEnable Order

The first was related to editor vs build rather than IL2CPP specifically, because the same issue occurred for Mono. The undefined order of Awake vs OnEnable in separate scripts resulted in a NullReferenceException in the built version of the application, but not in the editor. In this case, Awake was executing before OnEnable in the editor, and vice versa in the build.

// GameObjectA
private void Awake()
{
    _a = new A();
}

public void DoSomething()
{
    _a.DoSomething(); // exception, _a is null!
}

// GameObjectB
private void OnEnable()
{
    gameObjectA.DoSomething();
}

The quick fix in this case was wrapping _a in a property for lazy initialization instead of in Awake.

Marshal.SizeOf

The next exception to deal with was

ArgumentException: The t parameter is a generic type.

which was being thrown by a call to Marshal.SizeOf in the IL2CPP build, but not in the editor or Mono build. The specific line was in a binary reader class:

Marshal.SizeOf<HeaderSegment<TKey>>()

Other calls like Marshal.SizeOf<TKey>() work fine; it’s just not happy when T is generic.

I found it mentioned in a 2020 dotnet/runtime GitHub issue where someone commented it’s by design, but of course it doesn’t help explain the difference in behavior between Mono and IL2CPP. There was also a comment about using Unsafe.SizeOf, instead, so I gave that a try and it worked. Since I didn’t feel like adding another DLL at the moment, I just used the UnsafeUtility included in the Unity.Collections.LowLevel.Unsafe namespace, which is just a wrapper for a lot of the System.Runtime.CompilerServices.Unsafe methods, anyway.

A day or two later I did some research on these two SizeOf variations, which lead to resolving another related issue. I’ll cover that in the next post.

SQLite Stripping

The final issue only surfaced after I bumped up the managed stripping level to Medium (Minimal and Low were fine). The goal was to get up to the maximum level, High. I got a NullReferenceException on the following line when trying to do a read.

var columnName = Table.FindColumnWithPropertyName (mem.Member.Name).Name;

FindColumnWithPropertyName was returning null, so accessing the Name property resulted in the NRE. So now, I’ll walk you through the process I took to get to the root cause. Here’s an abbreviated class that resembles the table being queried:

public class LineInfo
{
    public string LineName { get; set; }
}

Why is FindColumnWithPropertyName returning null?

public Column FindColumnWithPropertyName (string propertyName)
{
    var exact = Columns.FirstOrDefault (c => c.PropertyName == propertyName);
    return exact;
}

Answer: The Columns property is empty here.

Why is the Columns property empty?

Columns is initialized in the TableMapping constructor but I noticed GetPublicMembers was returning an empty list.

var members = GetPublicMembers(type);

Why is GetPublicMembers returning an empty list?

private void GetPublicMembers()
{
    ...
    newMembers.AddRange(
        from p in ti.DeclaredProperties
        where !memberNames.Contains(p.Name) &&
            p.CanRead && p.CanWrite &&
            p.GetMethod != null && p.SetMethod != null &&
            p.GetMethod.IsPublic && p.SetMethod.IsPublic &&
            !p.GetMethod.IsStatic && !p.SetMethod.IsStatic
        select p);
    ...
}

Reflection…of course. This query gets all public instance properties that have both a getter and setter. But it turns out p.CanWrite was returning false and p.SetMethod was null, meaning there’s no setter.

Why is the setter missing?

This was a clear indication that it must being getting stripped, so I added a PreserveAttribute on the class, but it didn’t work. So then I tried applying the attribute to each property instead, and that did the trick. I thought applying it to the class meant it preserves all members, but apparently I was wrong. I found a GitHub issue that mentioned this exact stripping side effect: “IL2CPP Code stripping makes properties unwritable”.

public class LineInfo
{
    [UnityEngine.Scripting.Preserve]
    public string LineName { get; set; }
}

Conclusion

In the end, the goal of having a functional IL2CPP build of the application with managed stripping level set to High was achieved. Ensuring a successful build in CI is a good start, but actually running that build can save time down the road. This can even be Play Mode tests configured to run on the target platform.

It won’t be realistic for bigger projects, but still comes in really handy in certain cases, especially when prototyping and for small projects. SceneManager.sceneLoaded event can be enabled by enabling the “Reload Scene” checkbox.

Domain Reload On

Domain Reload Off

Font Weights

When you import a family of fonts, don’t explicitly reference them all in the TMP components. Instead, make the Regular font style reference all the variations.

Stylesheets

Create a stylesheet to add snippets for each weight (300, 400, etc.), or any other combination of markdown tags.

These will appear in the “Text Style” dropdown right below the text.

To get started, open the Asset menu and go to Create -> TextMeshPro -> Style Sheet

After customizing the stylesheet, set both the font and stylesheet as defaults in TextMeshPro PlayerSettings.

• compilation time every time you need to add or modify a log
• have to remember to remove them when finished (unless they’re useful enough to keep around)

Well, instead of putting those logs in the code, try moving them to a special kind of breakpoint known as a Tracepoint. Its primary use case is printing text to the Visual Studio console, but it also allows us to execute methods by putting the invocation inside a pair of curly braces. We can utilize this by calling Unity’s Debug.Log:

{Debug.Log("value: " + transform.name)}

You can include custom templates per project.

Without templates, one of the first things I do for many new scripts is remove the default Start and Update methods, along with the comments. In the attached screenshots I use an interface template and a “classes sealed by default” template.

The compiler complains about missing something called IsExternalInit. If you search online, you’ll find a small snippet of code that you can add to your project to fill in the missing piece.

Of course, MonoBehaviours can’t benefit from this feature, but still a lot of use cases. Immutable properties ftw!

init keyword docs

IsExternalInit snippet