QuickLaunch system status


About this page

Up-to-the-minute information on service availability. Check here for the status of service availability and/or incident report. (All dates and times are in Eastern Standard Time (EST).)

 
 

Status

  • QuickLaunch Services
    • Single Sign-On & Identity Operational
    • Automated Provisioning Operational
    • Password Manager with MFA Outage
    • Multi-Factor Authentication Outage
 

QuickLaunch Services Major Update: QuickLaunch Services Restoration Plan - Friday, August 26th 4 PM ET

QuickLaunch understands the impact of this Event and has been working around the clock to ensure all Services are restored, tested and available before the end of the weekend. QuickLaunch is targeting to complete its restore activity by early Saturday (8/27/22) morning (4 am ET). Thanks for your patience and understanding. Please bookmark the status page to stay up to date and create tickets for any issues you/your end users experience once the restore is complete so we can ensure its tracked to resolution.

QuickLaunch Service Restored/Updated through Plan of Action
QuickLaunch Service Plan of Action/Activity Description IDP and Single Sign-On Password Manager (PWMS) Adaptive MFA First Time Users (FTU) Custom Design & Branding Activity Date Activity Time Downtime Required Activity Status
Service Patches
User Access Manager 8.12.1 Patch API's timeouts reduced so that login page Request doesn't fail Yes No No No No 8/23/2022 3:45 PM ET Yes Completed
Admin 8.12.2 Patch This patch addresses blank page problem throwing script error while accessing thick client apps such as Microsoft Office 365 desktop apps (Teams, Outlook etc.) and few VPN clients. Yes No No No No 8/24/2022 5:15 PM ET No Completed
PWMS 8.12.3 Patch Improvements - Asynchronous calls to elastic search, reduced time-outs on HTTP calls. No Yes No No No 8/26/2022 3:00 AM ET No Completed
Restoration of Password Manager (PWMS), Adaptive Multi Factor Authentication(AMFA), First Time User (FTU) and Custom Design and branding
Phase 1: Migrating Elastic Search from dockerized cluster to AWS EC2 cluster Elastic Search on dedicated EC2 Cluster. No Yes Yes Yes No 8/26/2022 3:00 AM ET No Completed
Phase 2: Elastic Search data migration from dockerized cluster to AWS EC2 cluster Elastic Search data migration No Yes Yes Yes No 8/26/2022 4:30 AM ET No Completed
Phase 3: Configuring Elastic Search analytics and services to capture Elastic Search data into AWS EC2 cluster Elastic Search Configuration No Yes Yes Yes No 8/26/2022 4:30 AM ET No Completed
Phase 4: Scale PWMS AWS EC2 infrastructure and enhance PWMS API's to sustain high latency Performance improvement No Yes No No No 8/26/2022 5:00 AM ET No Patch Deployed- Ongoing activity - Week of Aug 29th
Phase 5: Re-enable PWMS, AMFA, FTU and Custom Design & Branding Complete QuickLaunch Services restoration No Yes Yes Yes Yes 8/27/2022 4:00 AM ET No Scheduled

Ongoing Incidents

  • QuickLaunch IDP8 - Investigating Service Disruption Alarms

    Lasted for 1 day
    Affected Service: Password Manager with MFA Multifactor Authentication August 24, Wed

    Monitoring

    QuickLaunch Single Sign-On thick client authentication patch released - Wednesday, Aug 24th - 6:00 PM ET

    QuickLaunch has deployed a patch to restore thick client authentication of desktop apps on Aug 24th at 5:15 PM ET. This patch addresses blank page problem throwing script error while accessing thick client apps such as Microsoft Office 365 desktop apps (Teams, Outlook etc.) and few VPN clients.

    Status Updates: We have received feedback that customers are not getting updates. Bookmark this page for status updates. https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    18:23, Aug 24 EDT

    Monitoring

    QuickLaunch Services - Monitoring Update - Wednesday, Aug 24th - 2:45 PM ET

    QuickLaunch Login services continue to be available without any alerts, interruptions or reported incidents. QuickLaunch support engineers are monitoring the services 24x7.

    Some customers and users are reporting blank page on login for new users which is not caused by service interruption. Its due to the patch (8.12.4) that was released 8/23/22 at 3:45 PM ET. This issue has been reported to QuickLaunch support. A patch is being developed to address the notification process for First Time Users (FTU). Will update once patch is developed with ETA for release.

    Status Updates: We have received feedback that customers are not getting updates. Bookmark this page for status updates. https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    16:09, Aug 24 EDT

    Monitoring
    8:45 AM ET

    QuickLaunch Services - Monitoring Update - Wednesday, Aug 24th - 8:45AM ET

    Please be advised that QuickLaunch Login services continue to be available without any alerts, interruptions or reported incidents. QuickLaunch support engineers are monitoring the services 24x7.

    QuickLaunch architects are still working on a plan for restoring Password, AMFA and Custom CSS services. The plan will be shared once Proof Of Concept(POC) is tested properly by QuickLaunch engineers. Revised ETA to be provided soon.

    Status Updates: We have received feedback that customers are not getting updates. Bookmark this page for status updates. https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    9:30, Aug 24 EDT

    Investigating
    5:45 PM ET


    Solution #3 - Denial of Service Patch Released for Restoring Login Services Update:

    QuickLaunch Login services continue to be available without interruption or reported incidents. QuickLaunch support engineers are monitoring services 24x7.

    QuickLaunch architects are still working on a plan for restoring Password, AMFA and Custom CSS services. ETA is 8/24. The plan will be shared once it has been prepared.

    Status Updates: We have received feedback that customers are not getting updates. Bookmark this page for status updates. https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    5:58, Aug 23 EDT

    Monitoring
    4:45 PM ET


    Solution #3 - Denial of Service Patch Released for Restoring Login Services - Update: -

    QuickLaunch has reported that in the last hour of the patch being released into production, login services continue to available without service interruption. Customers are being contacted to confirm availability and to record any custom configuration issues.

    QuickLaunch support engineers are now monitoring Login Services logs.

    QuickLaunch architects are working on a solution for restoring Password Management Services, AMFA Services and Custom CSS Services. Once this plan is outlined, it will be posted as an update.

    Status Updates: We have received feedback that customers are not getting updates. Bookmark this page for status updates. https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    4:47, Aug 23 EDT

Recent Incidents

  • QuickLaunch IDP8 - Investigating Service Disruption Alarms

    Resolved in 7 hours
    Affected Service: Single Sign-On & Identity Provider Password Manager with MFA Multifactor Authentication August 23, Tue

    Resolved
    3:45pm ET


    Solution #3 - Denial of Service Patch Released for Restoring Login Services -

    Major Update: QuickLaunch has deployed a patch for restoring login services (8.12.4) to production. This patch restores login services without password management and AMFA services as an emergency workaround. This patch will not call custom CSS pages and hence for customers with a customized login/post-login page, please be advised that you will see a more standard login page experience.

    Login services for affected customers should be restored, however there may be customers with custom configurations that will require scheduled calls with the QuickLaunch engineers.

    Solution #1 Denial of Service Workaround Update

    This solution continues to being deployed for customers. Customers who have deployed this solution have been able to restore login services for end users.

    Status Updates: We have received feedback that customers are not getting updates. Please go to save this page for updates: https://quicklaunch.io/outage-status/

    Thanks-QuickLaunch Support

    3:58, Aug 23 EDT

    Fixing
    2:15pm ET -


    Solution #3 - Denial of Service Patch Developed for Restoring Login Services

    QuickLaunch has developed a patch for restoring login services without password management and AMFA. This patch is being tested. QuickLaunch currently expects to complete testing by approximately 3:45pm ET. If tests are successful, we will send an update once the patch is scheduled to be released in production. Once released, this will restore login services for all affected customers.

    Solution #1 - Denial of Service Workaround Update

    This solution continues to being deployed for customers. Customers who have deployed this solution have been able to restore login services for end users.

    Thanks-QuickLaunch Support

    3:43, Aug 23 EDT

    Fixing


    Solution #1 Major Update - Denial of Service Workaround Successfully Deployed:

    QuickLaunch has tested solution #1 to workaround the denial of service attack on password management services successfully. QuickLaunch setup a SAML2.0 bridge for login services that bypasses the affected password manager services as well as AMFA services. In test, the login services successfully transact logins. The QuickLaunch team has applied this solution to two customers and this is running successfully in production. Our team will start reaching outbound to customers to queue up this option.

    Solution #2 Update - AWS:

    In addition to this, QuickLaunch architects are still working with AWS on resolving the denial of service attack. We have not yet received an ETA for this resolution.

    Thanks-QuickLaunch Support

    3:10, Aug 23 EDT

    Fixing

    What is the this outage, technically?

    Denial of Service on QuickLaunch Password Manager Services. This DOS attack on the public URL - Password.QuickLaunch.io is generating password reset by what seems to be a bot that is using one of the customer domains to make password reset calls. The flood of these password reset calls which started at approximately Tuesday, 8:15 AM ET are creating performance issues on Password Manager services and as the password reset calls pool, the logs show that the Login services become non-performant and unavailable as a result. Due to QuickLaunch’s architecture setup on multi-node and auto-scaling, the Login service will “spin” for end users as Login requests continue to increase and pool.
    Snippet of Denial of Service logs:
    Aug 23 13:01:36 QL8-PROD-PASSWORDMANAGER1 kernel: [ 913.933432] TCP: request_sock_TCP: Possible SYN flooding on port 8080. Sending cookies. Check SNMP counters.

    What is the impact of this outage?
    QuickLaunch Password manager is tightly coupled with identity infrastructure. As the Password manager API is under attack it has also impacted login process to become sluggish to the extent of being unavailable most times.

    Are all institutions affected or just us, or something in between?
    All schools using Password Manager v8.12 are affected.

    What is QuickLaunch doing at this time? what is the recommended plan of action?
    QuickLaunch is pursuing a two pronged attack at this point in time:
    a. QuickLaunch Network engineers and AWS support are on a call to address this denial of service attack on the QuickLaunch Password Manager services - Password.QuickLaunch.IO.
    b. QuickLaunch architects are setting up and testing emergency work arounds to enable Login services without calling Password Management services.

    Could you not block the client domain that is using the bot to make a password reset call?
    We cannot as we are a multi-tenant platform and request comes in from worldwide locations.

    Can we use the system without using password manager?
    Yes, This is part of work around solution till the time we are able to resolve Denial of Service attack.

    Is the attack coming from a specific location and IP address?
    The attack seems to be coming from different locations and IP addresses.

    How can we receive current updates to keep our community abreast?
    You can click on this link for most recent updates - https://quicklaunch.io/outage-status/. Additionally, you can also reach out to your QuickLaunch Customer Success managers and Support teams via email, ticket, and teams chat. QuickLaunch customer representatives are outreaching all affected clients individually too.

    Thanks-QuickLaunch Support

    12:57, Aug 23 EDT

    Fixing

    9:15am ET - Denial of Service on QuickLaunch Password Manager Services: QuickLaunch engineers isolated Password Manager services to its own multi-node architecture. Engineers identified in the logs of the Password Manager service a denial of service attack on the public URL - Password.QuickLaunch.io. The denial of service attack is generating password reset by what seems to be a bot that is using one of the customer domains to make password reset calls.

    The flood of these password reset calls which started at approximately 8:15am ET are creating performance issues on Password Manager services and as the password reset calls pool, the logs show that the Login services become non-performant and unavailable. Due to QuickLaunch’s architecture setup on multi-node and auto-scaling, the Login service will “spin” for end users as Login requests continue to increase and pool.

    QuickLaunch Network engineers and AWS support are on a call to address this denial of service attack on the QuickLaunch Password Manager services - Password.QuickLaunch.IO.

    Emergency Solutions: In parallel, QuickLaunch architects are setting up and testing emergency work arounds to enable Login services without calling Password Management services. One solution includes temporarily disabling Password Manager services until such time as the denial of service attack is resolved so that customers and end users can login and access services during this critical time.

    Thanks-QuickLaunch Support

    10:35, Aug 23 EDT

    Investigating
    Dear QuickLaunch Customers,

    8:15am - Our monitoring systems have received alerts of service degradation on QuickLaunch Password Manager and AMFA services. Currently the engineers are checking if these services are causing delays and timeouts in Login service load and processing. Login services may be intermittently available while engineers debug and troubleshoot.

    Thanks-QuickLaunch Support

    08:56, Aug 23 EDT

  • QuickLaunch IDP8 - Investigating Service Disruption Alarms

    Resolved in 16 hours
    Affected Service: Single Sign-On & Identity Provider Password Manager with MFA Multifactor Authentication August 23, Tue

    Resolved
    Please be advised that QuickLaunch services are back up and running to full capacity as of Monday Aug 22nd, 11:50 PM EST. Login services and all other modules should be functional for all clients and users. Support engineers are doing a comprehensive testing on all the services that affected AWS eLBR, database and docker services performance as we provide this update.

    Quicklaunch runs Quick Launch-Identity as authentication and authorization service in AWS and relies on docker swarm overlay network in conjunction with AWS internal DNS resolver for internal service discovery of microservices. We found an anomaly in docker network which was causing the service discovery to fail intermittently causing our password manager APIs to turn sluggish creating performance degradation to the point where services were unavailable.

    After multiple attempts to make both the networks play well together, team decided to do away with dockerized version of password manager to a more conventional horizontally scalable EC2 instance based service which fixed the service discovery issue and thereby the intermittent anomaly that was experienced during this period.

    We know how critical this service is to our clients, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability.

    We appreciate your patience while we worked through identifying, resolving and making the Identity services available again for all users.

    Thanks-QuickLaunch Support

    1:41, Aug 23 EDT

    Fixing
    9:15 PM EST - Engineers have begun troubleshooting the service calls that are being made between the eLBR, docker and databases during login and post-login process by introducing caching mechanism to impacted services (getAllMobileCarrier, getMultiFactorAuthenticationSettings, getVerificationSettingsForTenant) that seem to be causing the inability for logins to complete or to be delayed .

    Caching parameters are being revised to 10 minutes and testing is being done to see if this will address the
    unavailability of post-login services.

    Next update to be provided around 10:15 PM EST.

    21:23, Aug 22 EDT

    Fixing
    7:30 PM EST - Some of the API's responsible for rendering post-login are having performance delays for certain customers and users. Support engineers continue to work on the issues that are affecting AWS eLBR, database and docker services performance.

    Next update around 8:45 PM EST

    Thanks-QuickLaunch Support

    19:51, Aug 22 EDT

    Fixing
    Dear QuickLaunch Customers,

    Please be apprised on further constructive progress .

    6:00pm ET - Login services were restored for all customers. Some users are reporting post-login performance delays which are being investigated. Support engineers are continuing to review and monitor performance between the AWS eLBR, database and docker services.

    Next update by 7:15 PM EST.

    Thanks-QuickLaunch Support

    18:44, Aug 22 EDT

    Fixing
    Dear QuickLaunch Customers,

    Further to our last update, We are beginning to isolate each docker service, database and eLBR to identify what is causing the slow queries which are in turn making the services intermittently available or creating performance degradation to the point where services are unavailable.

    The secondary infrastructure is brought up and now we are starting to isolate services.

    As of now we don’t have an update for resolution. We have all-hands-on deck for this unprecedented issue.

    We shall provide the next update around 6:15 PM EST.

    Thanks-QuickLaunch Support

    17:42, Aug 22 EDT

    Fixing
    Dear QuickLaunch Customers,

    Quicklaunch would like to provide some additional information about the service degradation that we are experiencing as we provide this update. We understand the impact of the issues are critical and unprecedented and all QuickLaunch engineers and our senior support staff are working to get identity (authorization and authentication) services to 100% availability for all users.

    Enclosed below is a summary of what issues we are experiencing with our identity services and what actions have been taken thus far.

    Overview: QuickLaunch runs Quick Launch-Identity as authentication and authorization service in AWS.

    9:30 AM EST - QuickLaunch received alerts from our Network Operation Center team and reports from clients that users are unable to authenticate intermittently.

    9:35 AM EST - QuickLaunch started troubleshooting each of the identity services and identified that users were intermittently able to login while some attempts were failing.

    1:45 PM EST - QuickLaunch identity services restored resulting in intermittently successful logins.

    3:22 PM EST - Although 60% of users were successfully logging into services, QuickLaunch made the decision to run a full service restart of identity services as the docker network within the multi tenant AWS environment had been acting up and not serving downstream responses consistently through certain nodes hosted in AWS.

    3:40 PM EST - QuickLaunch initiated bringing up a secondary Identity environment.

    We will work on sending updates every 30 minutes as services restart. We anticipate this taking another 30-60 minutes.

    At this time QuickLaunch doesn’t have a specific root cause but is working with AWS to identify potential causes as we observed slow query logs that currently seem to be related to docker network, eLBR and the database services. As services are restarted, the root cause will be further investigated and shall be informed as part of the RCA once confirmed along with a immediate resolution plan. We will send another update at 5:15 PM ET.

    We know how critical this service is to our clients, their applications and end users, and their businesses. We will do everything we can to learn from this event and use it to improve our availability even further.

    Thank you for your patience at this time as we work through identifying, resolving and making the Identity services 100% available for all users.

    Thanks-QuickLaunch Support

    17:12, Aug 22 EDT

    Investigating
    Please be advised that we are still experiencing issues and we are still working on it.
    We shall keep you posted with further proceedings periodically followed by a detailed RCA.

    Thanks-QuickLaunch Support

    13:40, Aug 22 EDT

    Investigating
    Please be advised that we are continuing to address the problem on a warfront. We have identified the problem to be with our Identity system currently undergoing an extremely heavy user load. Our executive and Incident management teams are working very closely to resolve this problem asap. We shall keep you posted with further proceedings periodically followed by a detailed RCA.

    We appreciate your patience and support all this while.

    Thanks-QuickLaunch Support

    11:56, Aug 22 EDT

    Investigating
    We sincerely apologize for the service outage that you might be facing right now. We’re doing everything in our power to get things up and running as quickly as possible.

    We’re working around the clock to make sure you are able to access your account, sending out emails with updates on what we’re doing to restore functionality, and answering any questions you may have. We want you to know that your patience has not gone unnoticed.

    Our team is committed to making sure we provide the best possible service, and we take every opportunity to improve how things work behind-the-scenes. We are also reviewing our policies for better communication on ongoing issues.

    10:57, Aug 22 EDT

    Investigating
    We’re currently investigating reports of service disruption alarms on QuickLaunch IDP 8 services. Portal logins for institutions running on QuickLaunch 8 may be unavailable temporarily. Our team is investigating and we shall post another update in 30 minutes.

    09:36, Aug 22 EDT