Provide meta information for expected amount of certain elements for monitoring purposes

[amauch-adobe]

We have been looking at providing some generalized monitoring for the most trafficked pages on ACOM/BACOM milo pages, such as checking that all CTA buttons have displayed and rendered. Milo core has provided a hidden element (https://jira.corp.adobe.com/browse/MWPW-145591) that gets rendered at the very end of all sections, which give's us a good feeling that all sections loaded up, whether or not all blocks loaded (essentially checks that a page isn't blank), but we are looking to find a good way of monitoring general page elements to ensure specific content within blocks have loaded properly, i.e. CTA buttons.

We thought about it in a high level sense of grabbing all buttons based on a classnames and then iterating over them to ensure they have rendered, but this also creates a loop hole of not knowing how many buttons should be on that page. The test could falsely pass since what it found on the page may have rendered successfully, even though a block may have not fully decorated correctly due to a code change or authoring mishap. A failed block will show up blank on the page (in production) with other elements in other sections fully rendered. We are wanting to catch that issue when monitoring on production.

To help with this concept we were wondering if it were possible to have a meta value created that would provide stats on the page, for example how many blue buttons are expected to be on the page. That way we could find that element, pull the attribute value so we could then compare how many we actually find vs. how many are expected on the page. Is this something that Franklin should inherently provide or on a Milo side? Is this a plausible idea, or is there a better way?

We are asking this because other companies are going to run into our situation of needing to monitor things over numerous pages, etc. and not just us, but also because we want to get away from having to strictly tied elements that can easily change on the fly from authoring changes. In the past we have had CSOs fired erroneously because of an approved authoring change that removed a specific element we were looking for in a monitor test.

[mokimo]

Copy/Pasting my answer from our slacks:

The question of finding out how many elements there are, is pretty hard.

• Franklin provides the semantic markup and how many elements have been authored
• Milo has logic around styling and altering those elements. Each block decides itself on what it does with authored elements OR metadata
• Milo also has delayed elements such as the global navigation OR even further delayed items like a 3rd party integration such as the universal nav which renders a sign in button

I'm not aware of any straightforward way to tell how many elements should be on a page. Elements might also get hidden or shown optionally. Potentially only visible as you scroll due to intersection observers or loading on click.

I'd say the franklin markup might provide a first good guess on a minimum of elements, but blocks could still hide or delay those elements. Milo also has no insight into what 3rd party integrations put on the page

[overmyheadandbody]

MEP can also interfere here, which is a completely different stack that applies personalization based on multiple manifests and conditions.

So yes, this isn't at all straightforward.

Could RUM data help? They do have checkpoints for block loading and images/elements coming in view, maybe that could come in handy.

[thedoc31]

@mokimo @overmyheadandbody @davidnuescheler Let me reframe this a bit, because the question doesn't fully describe what we are trying to do or what our goal is.

Problem:
As an SRE team, we have to monitor critical pages on adobe.com to make sure they are loading properly and providing users the correct experience. Historically for AEM, we had a PAGELOADOK that was added to the footer of the page, and if that wasn't there, we knew it didn't draw properly and could alert.

Now with Franklin/Milo pages, that has changed because the methodology of drawing pages is fundamentally different. We have tried to set monitors on page elements, forms, blocks, and buttons, but because the content team is constantly changing pages, elements, and blocks or running A/B tests, this generates false alarms like crazy. The toil trying to keep everything up-to-date is untenable for the long-term.

We need a generic or automatic way to determine that not only has the DOM loaded, but that all elements on the page that are supposed to load actually did. If they didn't, then we need to CSO.

Current state:
We are only able to validate that the DOM actually loaded and we do not have a blank page.

Goal:
Based on Adobe Analytics data, automatically monitor all top-level product pages and the top 100 pages on adobe.com based on a combination of page hits weighted by the average revenue generated per page. If the monitors fail, we automatically throw a CSO.

** Question **
How would you recommend we approach this? Is this something we need to bring up with the Franklin team as a product enhancement, or can we come up with a Milo-based solution that will suffice?