RCA - March 15, Friday QuickLaunch - Investigating Service Disruption Alarms

RESOLVED - Service Degradation
Date: 03/15/2024
Time: 11:03 AM ET

Logins have been restored and all services are working smoothly.

Identified - Service Degradation
Date: 03/15/2024
Time: 10:45 AM ET

QuickLaunch logins are giving 500 error, the technical team is working on it. We will provide an update shortly.

RCA - March 02, Saturday QuickLaunch - Investigating Service Disruption Alarms

RESOLVED - Service Degradation
Date: 03/02/2024
Time: 10:02 PM ET

Please be advised that we received reports of a potential performance degradation on QuickLaunch-IDP services from our monitoring devices.

QuickLaunch has mitigated the issue and the services have been restored to normalcy at 10:02 PM ET.

---

RCA - February 08, Thur QuickLaunch IDP8 - Investigating Service Disruption Alarms

RESOLVED - Service Degradation
Date: 02/08/2024
Time: 2:32 PM ET

QuickLaunch service has been fully restored. Please report any issues to our support team if you are still experianceing any issues. RCA will be available as soon as possible.

Identified - Service Degradation
Date: 02/08/2024
Time: 2:26 PM ET

Some schools are reporting login failures as redirection to post login is not generating results. QuickLaunch is working to restore normal operations ASAP. We will continue to update status page with updates or within one hour from the last update. Please watch status page for the most recent updates.

Identified - Service Degradation
Date: 02/08/2024
Time: 1:15 PM ET

Few schools are reporting login failures as redirection to post login is not generating results. QuickLaunch is working to restore normal operation and will continue to provide timely updates.

RESOLVED - Service Degradation
Date: 02/08/2024
Time: 10:15 AM ET

We were able to restore slow logins at 10:15. The clients facing intermittent issues post resolution is now fully resolved. QuickLaunch is continuously monitoring the systems to ensure there is no further impact on services. We will share RCA to impacted customers in the next 24 hours.

Identified - Service Degradation
Date: 02/08/2024
Time: 10:05 AM ET

QuickLaunch is currently investigating slow logins in excess of 30 seconds. QuickLaunch is working to restore normal operation and will continue to provide timely updates.

---

RESOLVED - Service Degradation
Date: 01/22/2024
Time: 11:29 AM ET

Because of our multi-node architecture, the performance degradation on slow logins QuickLaunch experienced this morning was limited. For those customers that were impacted, Service is now fully restored. QuickLaunch will be providing RCA's to the customers that were impacted.

Identified - Service Degradation Date: 01/22/2024
Time: 11:19 AM ET

QuickLaunch is currently investigating slow logins in excess of 30 seconds. QuickLaunch is working to restore normal operation and will continue to provide timely updates.

RESOLVED- Service Degradation
Date: 12/04/2023
Time: 10:44 AM ET

Hello,

I'm pleased to announce that the service degradation affecting our Product's Single Sign-On & Identity, and Password Manager with MFA has been successfully resolved. Our dedicated team worked diligently to address the issues promptly, and I'm grateful for your patience throughout this process.

All services are now fully restored, ensuring a seamless experience for our users. We understand the importance of these features to your operations, and we appreciate your understanding during the resolution period.

If you encounter any lingering concerns or have additional questions, please do not hesitate to reach out to our support team or your Customer Success Manager.

Your satisfaction is our priority.

Thank you for your understanding and continued trust in our services.

Sincerely,
Jim Knutson – VP Customer Success – QuickLaunch
M:+1 305 916 0844
E: Jim_Knutson@Quicklaunchsso.com

Service Degradation Alarm
Date: 12/04/2023
Time: 10:08 AM ET

Hello,

We are currently experiencing a service degradation affecting our Product's Single Sign-On & Identity, and Password Manager with MFA. Our team is dedicated to swiftly resolving these issues, and we expect full restoration soon. I will continue to provide updates until the disruption is resolved. We appreciate your patience. For any concerns, please contact our support team or your Customer Success Manager.

Alternatively feel free to reach me directly on 1 305 916 0844 or Jim_Knutson@Quicklaunchsso.com at any time.

Sincerely,
Jim Knutson – VP Customer Success – QuickLaunch
M:+1 305 916 0844
E: Jim_Knutson@Quicklaunchsso.com

RESOLVED - Delay in AWS SMS for MFA
Date: 09/20/2023
Time: 8:45 AM ET

We completed the SMS upgrade, testing, and production rollout early this morning. We are seeing ZERO failures on SMS at this time. In addition to resolving this issue we have made the QuickLaunch product much more robust in the following ways:

• Upgraded the product to allow for additional SMS vendors.
• Added Additional SMS vendors.
• Added real time monitoring and alerting of SMS message failures.
• Multiple SMS vendors on hot standby.
• Added failover process to ensure minimal disruption in the future.

We consider this issue resolved. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issues. Thank you!

As always, I am available for any of our customers at any time, please do not hesitate to reach out.

Sincerely,
Jim Knutson – VP Customer Success – QuickLaunch
M:+1 305 916 0844
E: Jim_Knutson@Quicklaunchsso.com

Delay in AWS SMS for MFA
Date: 09/19/2023
Time: 5:00 PM ET

Hello,

I have confirmed a resolution in our testing and QA environments. Production deployment will be proceeding TONIGHT after close of business.

I will keep posting further updates until this process is completed, and the issue is completely resolved. Thank you for your patience. My next update will be 9 AM EST Sept 20th. Contact our support team or your Customer Success Manager for any issues.

Alternatively feel free to reach me directly on 1 305 916 0844 or Jim_Knutson@quicklaunchsso.com at any time.

Jim Knutson - Vice President Customer Success - QuickLaunch

Delay in AWS SMS for MFA
Date: 09/19/2023
Time: 12:00 PM ET

Hello,

While SMS failures have been reduced, we are still seeing some intermittent failures. We are working hard to resolve this by the close of business today.

I will keep posting further updates untill this process is completed and the issue is completely fixed. My next update will be 5 PM EST Sept 19th. Please reach me directly at 1 305 916 0844 or Jim_Knutson@quicklaunchsso.com or your Customer Success Manager for any issues.

Jim Knutson Vice President Customer Success - QuickLaunch

Delay in AWS SMS for MFA
Date: 09/13/2023
Time: 12:00 PM ET

We are still seeing some intermittent SMS delivery failures and are planning to bring on an additional provider this weekend to overcome these limitations without disruption.

Additionally, we have added redundancy to these systems to eliminate a single point of failure going forward. This will eliminate the delays or failures of SMS messages being faced now.

As previously stated: It is recommended that Email OTPs be used as an alternative for SMS OTPs for the New User Registration process. For other MFA verification, users can use any available alternate factor such as Email, Security Questions, Authenticators, Yubikey or Mobile App etc.

We will keep posting further updates till this process is completed and the issue is completely fixed. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issues. Thank you!

Delay in AWS SMS for MFA
Date: 09/07/2023
Time: 12:20 PM ET

We have found that the issue of SMS OTPs being delayed or not delivered is due to regulatory changes in the way the SMS delivery infrastructure works on the Telecom networks. The number of SMS messages that can be sent in a particular time duration has been limited due to these regulatory changes in the Telecom Networks. This is causing delays or failure in delivery of SMS OTPs. QuickLaunch has now commissioned an additional SMS delivery system to overcome this limitations. The additional SMS delivery system has been tested and implemented. There is a process required for approval by regulatory authorities for activation of the additional SMS delivery service to start sending out SMS messages. QuickLaunch has also completed the process of seeking approval and is awaiting approval from the regulatory authorities. Upon approval, the additional SMS delivery system will also start sending SMS OTPs in addition to the current SMS delivery system. This will eliminate the delays or failures of SMS messages being faced now. The regulatory authorities have informed QuickLaunch that the process of approval will take 2 - 3 days. QuickLaunch is actively following up to expedite this process.

In the meanwhile, it is recommended that Email OTPs be used as an alternative for SMS OTPs for the New User Registration process. For other MFA verification, users can use any available alternate factor such as email, security questions, Authenticators, Yubikey or Mobile App etc.

We will keep posting further updates till this process is completed and the issue is completely fixed. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issues. Thank you!

Delay in AWS SMS for MFA
Date: 09/05/2023
Time: 3:00 PM ET

We are continuing to work with our SMS service provider (AWS) for a fix on this issue. AWS has not confirmed a time when this issue will be fixed and they are working with Telecom networks to get this issue fixed. In the meanwhile, we have tested an alternate SMS services provider as an alternative. The test results show that the tests have been successful. We are proceeding to integrate the alternate SMS service provider as a backup SMS delivery service when AWS SMS delivery faces issues.

We will update once the alternate SMS service provider is integrated and is able to deliver SMS messages for MFA. Keeping in mind that we are in a peak season for usage, we are taking all precautions to ensure that this change does not disturb the system in any way. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issues. Thank you!

Delay in AWS SMS for MFA
Date: 09/05/2023
Time: 11:30 AM ET

We have been in touch with the SMS service provider (AWS) regarding the delays in delivery of SMS messages for MFA. AWS is continuing to work on fixing the issue and as an alternative we are currently working with another SMS provider as an alternate method of sending SMS messages.

We will provide another update by 3 PM ET. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issues being faced. Thank you!

Delay in AWS SMS for MFA
Date: 08/31/2023
Time: 10:00 AM ET

We have been informed by our SMS provider (AWS) about them getting reports of delay in receiving SMS OTPs for T-Mobile and AT&T users, due to changes by AWS in their delivery mechanism of SMS to these providers. If you are facing these delays, please use any available alternate factor (email, security questions, Authenticators, Yubikey or Mobile App etc.) for your MFA.

We are monitoring this with AWS and will keep this page updated with the latest information. Please reach out to support@quicklaunchsso.com or your Customer Success Manager for any issue being faced.

QuickLaunch SRP Saturday, August 27th, 5 AM EST

QuickLaunch SRP Friday, August 26th 4 PM ET

August 24, Wed QuickLaunch IDP8 - Investigating Service Disruption Alarms

August 23, Tue QuickLaunch IDP8 - Investigating Service Disruption Alarms

August 22, Mon QuickLaunch IDP8 - Investigating Service Disruption Alarms