CaaS Link Truncation

[meganthecoder]

Problem:
During bulk updates on bacom, some of the caas links got truncated because they were too long (over 2000 characters). This broke the caas cards on those pages and required manual authoring to fix.

Background:
From what I can tell, we had an issue with truncation during the loc process last year, and @chrischrischris implemented gzip compression for caas links in Oct 2023. PR #1401. Ticket: MWPW-135873

However, even with this compression, we have links over 2000 characters. For example, the caas link on the bacom resource page is 2940 characters. A lot of bacom's new resource topic pages are over the limit because they have a lot of customization.

Ideas on how to solve:

• Is there a better way to compress? Changing the compression format from gzip to deflate-raw will only reduce the length by about 2%, which isn't enough.
• Do we have to include the entire state in the config url or could we include only what options changed from the default?
• Could we shorten the param names?
  The minimum caas url length is 1247 characters (with the default options). If the Microsoft Word max is 2077, that means that we're using more than half of our space with boilerplate. Generally, I'm on the side of making names and labels human-readable, but in this case, it might be worth it to abbreviate the param names to save space.
• Other ideas?
• Also, how do we keep this from happening again in the future? Do we need a cap on the number of caas fields we add?

To note: We're also looking into this on the Microsoft side, but considering how slow that process can be, we're focusing on changes we can make in milo.

[jedjedjedM]

What exactly is limiting us to 2k chars? Have we taken this limitation to MSFT via support case or feature request? Akamai supports URLs in the 8k range and chrome supports up to 2mb.

Brotli compression will help a bit, but still probably not enough (18% savings with my test-case).

For those not deep into milo/caas link lore...
The issue with the milo links is that they represent the state of the configuration tool and are not the final link required by caas. The links in milo are a json object of the configuration tool gzip'd and base64 encoded. These are converted on the fly into a real caas link which is at least half the size. (~1100 chars for the resource center.)

If the caas config tool can build its state using a real caas link instead of a json object, that would probably be the ideal fix. @chrischrischris, do you think that is feasible?

[meganthecoder]

What exactly is limiting us to 2k chars? Have we taken this limitation to MSFT via support case or feature request?

We believe this is a Microsoft limitation. We noticed truncation happening on links around 2000+ characters, and in Chris' ticket he says the Microsoft Word max is 2077. I'm not sure exactly where he got that number, but it's consistent with what we saw.

We want to open a ticket with Microsoft about this, but we need some kind of session id for them to track when the issue happened. (That's what they requested when we opened a ticket about the separate link shuffling issue). We need the id from @garvitjoshi so I'll follow up with him on if we have that.

[meganthecoder]

Update: Sunil is going to bring this up in his meeting with Microsoft

[sanrai]

Adding a few more possible solutions here.

A combination of these fixes will definitely fix/resolve the issue.

I listed these in terms of easiest to hardest to implement (and left other possible solutions out), but relatively, most of these fixes are quite simple.

1. Use different base than 64.
   Using base 85 for example should increase our upper bound by a few extra hundred characters

2. Split the base64 string:
   We could split the base64 encoded JSON into multiple parts, each under 2K characters. Then we reconstruct the full base64 string before decoding and parsing it.

For example:

[part1]base64-data-part-1
[part2]base64-data-part-2
[part3]base64-data-part-3

3. Shorten the URLs and then have some mapping to their encoded values
   We can create a hash of the config object into an id and store some mapping (either as a word table in the simplest case or some external store) and then pull from there.

[meganthecoder]

Thanks for the suggestions! A couple thoughts:

1. I don't see a standard web method for encoding/decoding ascii85. Do you know of one? If not, we'd need a third party library or something, which might have performance impacts.
2. Not sure I understand this idea. Are you proposing that we'd author two or three caas links in a row that all combine to make one card collection?
3. I'd like to avoid something like this if we can because of the performance impact and authoring overhead, but it's something to consider if nothing else pans out.

[sanrai]

Just an FYI since it's not inherently obvious -- compression algorithms like gzip already do this internally as part their algorithm and us doing this would give us no gain.

Could we shorten the param names? Generally, I'm on the side of making names and labels human-readable, but in this case, it might be worth it to abbreviate the param names to save space.

[meganthecoder]

I already tested this and got a small reduction of about 30 characters when abbreviating 4 or 5 params, so I consider it a viable option. It was only a 2.5% reduction though so it won't solve the problem on its own.

[sukamat]

@meganthecoder I believe I mentioned this in one of the earlier meetings related to this topic, but I am reiterating here for clarity.

The character limitation issue can be consistently reproduced in the MS Word Desktop app by inserting a long URL and re-opening the link edit window. Upon doing so, the URL appears truncated. It does not seem that a session ID is required to replicate this behavior.

Based on my previous discussions with Microsoft, they tend to prioritize addressing issues in the MS Word Web app rather than the Desktop version. In most cases, their focus is on ensuring that the Web app functions similarly to the Desktop app.

If you haven't already contacted Microsoft, I can bring this matter up during one of the bi-weekly meetings I attend with them, to explore the possibility of having this issue resolved.

Generally, I'm on the side of making names and labels human-readable, but in this case, it might be worth it to abbreviate the param names to save space.

Do you have any insights on the expected gains from this? I assume this could be manually tested by modifying the CaaS JSON and applying encoding/gzipping techniques.

Alternatively, maintaining a CaaS link map with Excel/JSON as a database might be a viable solution. Another approach could be a server-side solution (perhaps an AdobeIO endpoint) that uses a hashmap to return the CaaS link based on an ID. However, this would require modifications to multiple modules that handle CaaS to accommodate the change, not to mention the additional hop the page would need to make before getting the CaaS cards.

[meganthecoder]

• RE Microsoft: I wasn't in that discussion so thanks for explaining. I synced with Derron and team, and we haven't created a ticket yet. If you could bring this up with Microsoft in your next meeting, that would be great! @jedjedjedM 's point that the industry standard allows for much longer links seems like a good case for Microsoft to increase their limit. We understand that this might not be a high priority for them though so we'll move forward with what changes we can make in milo.
• RE Param names: I did a small test (see my response to Sanjay), but I can do a test with all params to get more accurate numbers.
• RE Excel/IO: This is definitely a possibility but I'd prefer to exhaust all options first for performance reasons.

[nerrdo]

To Jed's point, I do think that there is a case to be made to Microsoft that imposing such a tight character limit seems arbitrary from the customer point of view. Given that the potential exists for much larger URLs, it hardly seems unreasonable to suggest they increase it. But as Sunil points out, we need to be prepared for the fact that we're most likely asking them to change it in two products, even though we only care about one of them.

[chrischrischris]

Other than microsoft fixing the issue, we'll always have problems when many filters and/or tags are used. Things that might help:

Low Effort:

1. When converting the state into the hash, all of the properties that are using default values could be removed first.
2. Property names can be abbreviated in the state
3. Booleans can be converted to 0/1
4. Could aggregate properties into an array. E.g instead of having:

sortDateAsc: true,
sortDateDesc: true,
sortDateModified: false,
sortEnablePopup: true,
sortEnableRandomSampling: false,

we could do:

sortOpts: [1,1,0,1,0]

Medium Effort:

1. Similar to what @sanrai suggested: Instead of converting the gzipped data to a base64 string, convert it to a unicode string instead.
   This is somewhat non-standard, but all modern browsers support unicode characters so I don't think it'd be a problem.
   We'd use something similar to:

function arrayBufferToUnicodeString(buffer) {
    const uint16Array = new Uint16Array(buffer);
    let unicodeString = '';
    for (let i = 0; i < uint16Array.length; i++) {
        unicodeString += String.fromCharCode(uint16Array[i]);
    }
    return unicodeString;
}

function unicodeStringToArrayBuffer(unicodeString) {
    const uint16Array = new Uint16Array(unicodeString.length);
    for (let i = 0; i < unicodeString.length; i++) {
        uint16Array[i] = unicodeString.charCodeAt(i);
    }
    return uint16Array.buffer;
}

The unicode string would then have to be added to the url as a hash manually and NOT using new URL since that will encode the string
Quick tests show a nearly 2/3 length reduction of the string.
Issues: Need to make sure this will work in major browsers. I just did a quick test and am not 100% sure it will work for all data yet

High Effort:

1. Have the backend use id's for tags and filters so that we don't have to use the full string everytime they are specified. This will require backend changes as well as a way to look up the id/tag mapping (only for the configurator)

[meganthecoder]

We'll start with low effort item 1 removing defaults and see where that gets us https://jira.corp.adobe.com/browse/MWPW-158480

[meganthecoder]

Update: Removing defaults reduces most of our Batch 2 use cases enough except for the longest one (2940 characters on the main resource page), so we'll need something else as well. I tried low effort # 3, but that didn't change the length at all. I'll continue to try out these options and see what works.