Set starting send bitrate to 1000kbps to reduce time of low resolutio…

…n at start of connection (#2987)

CHANGELOG.md

@@ -5,32 +5,22 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [3.26.0] - 2024-10-07
 
 ### Added
 
 ### Removed
 
 ### Changed
 
+- Set starting send bitrate to 1000kbps to reduce time of low resolution at start of connection
 - Add per-frame encoding time in video encoding CPU connection health policy
 
 ### Fixed
 
-- Prevent DataMessage callback errors from killing a meeting
-
-## [3.26.0] - 2024-10-07
-
-### Added
-
-### Removed
-
-### Changed
-
-### Fixed
-
 - Avoid breaking audio input when external devices are disconnected on iOS browsers when using Web Audio by suspending and resuming the audio context in that case.
 - Fixed incoming audio loss calculation when server side network adaption and redundant audio features are running together.
+- Prevent DataMessage callback errors from killing a meeting
 
 ## [3.25.0] - 2024-09-10
 

docs/classes/setlocaldescriptiontask.html

@@ -123,7 +123,7 @@ <h3>constructor</h3>
 							<aside class="tsd-sources">
 								<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#constructor">constructor</a></p>
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L16">src/task/SetLocalDescriptionTask.ts:16</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L15">src/task/SetLocalDescriptionTask.ts:15</a></li>
 								</ul>
 							</aside>
 							<h4 class="tsd-parameters-title">Parameters</h4>
@@ -154,7 +154,7 @@ <h3><span class="tsd-flag ts-flagProtected">Protected</span> task<wbr>Name</h3>
 					<aside class="tsd-sources">
 						<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#taskname">taskName</a></p>
 						<ul>
-							<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L14">src/task/SetLocalDescriptionTask.ts:14</a></li>
+							<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L13">src/task/SetLocalDescriptionTask.ts:13</a></li>
 						</ul>
 					</aside>
 				</section>
@@ -172,7 +172,7 @@ <h3>cancel</h3>
 							<aside class="tsd-sources">
 								<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#cancel">cancel</a></p>
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L22">src/task/SetLocalDescriptionTask.ts:22</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L21">src/task/SetLocalDescriptionTask.ts:21</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -288,7 +288,7 @@ <h3>run</h3>
 							<aside class="tsd-sources">
 								<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#run">run</a></p>
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L32">src/task/SetLocalDescriptionTask.ts:32</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L31">src/task/SetLocalDescriptionTask.ts:31</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">

docs/classes/videocodeccapability.html

@@ -215,14 +215,15 @@ <h3>fmtp<wbr>Line<wbr>Matches</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L37">src/sdp/VideoCodecCapability.ts:37</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L50">src/sdp/VideoCodecCapability.ts:50</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
 								<div class="lead">
 									<p>Returns whether the codec capability fmtp line matches. This will not
 										attempt to match H.264 profile levels (e.g. 5.2, 3.1), see internal comments for
-									more detailed information.</p>
+										more detailed information. This function takes care of checking for an
+									expected payload type as well.</p>
 								</div>
 								<p>This function is meant to only be used internally.</p>
 							</div>
@@ -249,7 +250,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> av1</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L194">src/sdp/VideoCodecCapability.ts:194</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L221">src/sdp/VideoCodecCapability.ts:221</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -271,7 +272,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> av1<wbr>Main</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L184">src/sdp/VideoCodecCapability.ts:184</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L211">src/sdp/VideoCodecCapability.ts:211</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -293,7 +294,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> from<wbr>Signaled</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L201">src/sdp/VideoCodecCapability.ts:201</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L228">src/sdp/VideoCodecCapability.ts:228</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -321,7 +322,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L158">src/sdp/VideoCodecCapability.ts:158</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L185">src/sdp/VideoCodecCapability.ts:185</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -343,7 +344,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Baseline<wbr>Pro
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L81">src/sdp/VideoCodecCapability.ts:81</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L108">src/sdp/VideoCodecCapability.ts:108</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -367,7 +368,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Constrained<wbr>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L95">src/sdp/VideoCodecCapability.ts:95</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L122">src/sdp/VideoCodecCapability.ts:122</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -391,7 +392,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Constrained<wbr>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L147">src/sdp/VideoCodecCapability.ts:147</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L174">src/sdp/VideoCodecCapability.ts:174</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -415,7 +416,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>High<wbr>Profile
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L133">src/sdp/VideoCodecCapability.ts:133</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L160">src/sdp/VideoCodecCapability.ts:160</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -446,7 +447,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Main<wbr>Profile
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L110">src/sdp/VideoCodecCapability.ts:110</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L137">src/sdp/VideoCodecCapability.ts:137</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -471,7 +472,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp8</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L68">src/sdp/VideoCodecCapability.ts:68</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L95">src/sdp/VideoCodecCapability.ts:95</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -493,7 +494,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp9</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L177">src/sdp/VideoCodecCapability.ts:177</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L204">src/sdp/VideoCodecCapability.ts:204</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">
@@ -515,7 +516,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp9<wbr>Profile0</h3>
 						<li class="tsd-description">
 							<aside class="tsd-sources">
 								<ul>
-									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L166">src/sdp/VideoCodecCapability.ts:166</a></li>
+									<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L193">src/sdp/VideoCodecCapability.ts:193</a></li>
 								</ul>
 							</aside>
 							<div class="tsd-comment tsd-typography">

src/sdp/SDP.ts

@@ -927,8 +927,9 @@ export default class SDP {
         // `x-google-start-bitrate` is an unofficial flag that has existed in libwebrtc since its release and is unlikely
         // to be removed without notification.
         //
-        // libwebrtc reference: https://webrtc.googlesource.com/src/+/b6ef1a736ee94d97cc28f3bd59b826c716a3278f/media/base/media_constants.cc#97
-        const newLine = line + `;x-google-start-bitrate=${bitrateKbps * 1000}`;
+        // libwebrtc constant: https://webrtc.googlesource.com/src/+/b6ef1a736ee94d97cc28f3bd59b826c716a3278f/media/base/media_constants.cc#97
+        // libwebrtc parsing:  https://webrtc.googlesource.com/src/+/61c5e86dca59b0ac8240444e7df5c31da27ee40f/media/engine/webrtc_media_engine.cc#172
+        const newLine = line + `;x-google-start-bitrate=${bitrateKbps}`;
         dstLines.push(newLine);
       } else {
         dstLines.push(line);

src/sdp/VideoCodecCapability.ts

@@ -18,19 +18,32 @@ export default class VideoCodecCapability implements Eq {
   ) {}
 
   equals(other: this): boolean {
-    return (
-      other !== undefined &&
-      this.codecName === other.codecName &&
-      this.codecCapability.mimeType === other.codecCapability.mimeType &&
-      this.codecCapability.clockRate === other.codecCapability.clockRate &&
-      this.codecCapability.sdpFmtpLine === other.codecCapability.sdpFmtpLine
-    );
+    if (other === undefined) {
+      return false;
+    }
+
+    if (
+      this.codecName !== other.codecName ||
+      this.codecCapability.mimeType !== other.codecCapability.mimeType ||
+      this.codecCapability.clockRate !== other.codecCapability.clockRate
+    ) {
+      return false;
+    }
+
+    const thisFmtpLine = this.codecCapability.sdpFmtpLine
+      ? this.cleanupFmtpLine(this.codecCapability.sdpFmtpLine, this.codecName)
+      : '';
+    const otherFmtpLine = other.codecCapability.sdpFmtpLine
+      ? this.cleanupFmtpLine(other.codecCapability.sdpFmtpLine, other.codecName)
+      : '';
+    return thisFmtpLine === otherFmtpLine;
   }
 
   /**
    * Returns whether the codec capability fmtp line matches. This will not
    * attempt to match H.264 profile levels (e.g. 5.2, 3.1), see internal comments for
-   * more detailed information.
+   * more detailed information. This function takes care of checking for an
+   * expected payload type as well.
    *
    * This function is meant to only be used internally.
    */
@@ -40,7 +53,28 @@ export default class VideoCodecCapability implements Eq {
       return false;
     }
 
-    if (this.codecName === 'H264') {
+    const prefix = `a=fmtp:${expectedPayloadType} `;
+    if (!line.startsWith(prefix)) {
+      return false;
+    }
+    const fmtpLineFromSdp = line.substring(prefix.length);
+
+    return (
+      this.cleanupFmtpLine(fmtpLineFromSdp, this.codecName) ===
+      this.cleanupFmtpLine(codecCapabilityFmtpLine, this.codecName)
+    );
+  }
+
+  /**
+   * Helper function to clean up the FMTP line by removing unnecessary parameters
+   * and normalizing certain codec-specific attributes.
+   */
+  private cleanupFmtpLine(fmtpLine: string, codecName: string): string {
+    // Remove starting bitrate parameter which isn't relevant to matching.
+    const bitrateRegex = /;x-google-start-bitrate=\d+/g;
+    let cleanedLine = fmtpLine.replace(bitrateRegex, '');
+
+    if (codecName === 'H264') {
       // Given that most H.264 decoders for the past decade can support level 5.2, we do not bother checking the Profile Level
       // with regards to canonically defined, meeting wide, H.264 profile options (i.e. we intentionally do not split up `h264HighProfile`
       // into `h264HighProfileLevel52, `h264HighProfileLevel31`, etc.).
@@ -49,17 +83,10 @@ export default class VideoCodecCapability implements Eq {
       //
       // If maximum compatability with H.264 is desired by a builder they should stick to H.264 Constrained Baseline Profile.
       const profileLevelIdRegex = /profile-level-id=([0-9a-f]{4})[0-9a-f]{2}/i;
-      const modifiedCodecCapabilityFmtpLine = codecCapabilityFmtpLine.replace(
-        profileLevelIdRegex,
-        'profile-level-id=$1'
-      );
-      const modifiedLine = line.replace(profileLevelIdRegex, 'profile-level-id=$1');
-
-      return modifiedLine.startsWith(
-        `a=fmtp:${expectedPayloadType} ${modifiedCodecCapabilityFmtpLine}`
-      );
+      cleanedLine = cleanedLine.replace(profileLevelIdRegex, 'profile-level-id=$1');
     }
-    return line.startsWith(`a=fmtp:${expectedPayloadType} ${codecCapabilityFmtpLine}`);
+
+    return cleanedLine;
   }
 
   /**

src/task/SetLocalDescriptionTask.ts

@@ -3,7 +3,6 @@
 
 import AudioVideoControllerState from '../audiovideocontroller/AudioVideoControllerState';
 import DefaultBrowserBehavior from '../browserbehavior/DefaultBrowserBehavior';
-import DefaultModality from '../modality/DefaultModality';
 import SDP from '../sdp/SDP';
 import BaseTask from './BaseTask';
 
@@ -56,16 +55,20 @@ export default class SetLocalDescriptionTask extends BaseTask {
       sdp = new SDP(sdp).removeH264SupportFromSendSection().sdp;
     }
 
-    // We set content hint to `motion` as a workaround for the issue Chrome cannot enable temporal
+    // The default start bitrate is 300kbps. While this is a reasonable starting point with respect
+    // to network adaptation, the encoding bitrate does not adjust as quickly as the estimate, which
+    // can quickly max out on a clear network after a few seconds. Thus it will often take 10-25 seconds
+    // for the encode bitrate/resolution to reach the intended value. The network adaptation detects issues quick enough
+    // that raising it to 1000kbps is not much concern, on most networks the congestion controller
+    // will react to network restrictions before the first keyframe is even generated!
+    //
+    // Note that this is also important with regards to content share with SVC enabled,
+    // as we set content hint to `motion` as a workaround for the issue that Chrome cannot enable temporal
     // scalability for screen share https://bugs.chromium.org/p/chromium/issues/detail?id=1433486
     // As a side effect, content share may start at a low resolution and take a long time to adapt,
-    // especially when there is limited motion on screen. To mitigate the problem, we set a starting
-    // bitrate of 100 kbps for content share with SVC enabled.
-    const attendeeId = this.context.audioVideoController.configuration.credentials.attendeeId;
-    const isContent = new DefaultModality(attendeeId).hasModality(DefaultModality.MODALITY_CONTENT);
-    if (isContent && this.context.audioVideoController.configuration.enableSVC) {
-      sdp = new SDP(sdp).withStartingVideoSendBitrate(100).sdp;
-    }
+    // especially when there is limited motion on screen, if we do not set this slightly higher starting
+    // bitrate.
+    sdp = new SDP(sdp).withStartingVideoSendBitrate(1000).sdp;
 
     if (
       this.context.videoSendCodecPreferences !== undefined &&