AWS Partner Network (APN) Blog
Transforming Order Fulfillment with Event-Driven Architecture: A Rapid7 Success Story
By Art Smorodin, Senior Application Engineer – Rapid7
By Kashif Mohammed, Senior Application Engineer – Rapid7
By Priya Dodwad, Senior Manager Software Engineering – Rapid7
By Steven Warwick, Senior Solutions Architect – AWS
 |
| Rapid7 |
 |
Introduction
Rapid7, a leader in extended risk and threat detection, embarked on an exciting journey to transform their software order fulfillment process. By adopting an event-driven architecture, the company aimed to enhance its internal user experience, streamline operations, and unlock new levels of efficiency.
The key motivation behind this transition was the pursuit of a real-time and seamless approach to order fulfillment. The goal of this approach was to alleviate the burden on the internal team as well as provide a more cost-effective solution for order fulfillment.
Rapid7’s previous approach involved a middleware solution that employed a polling point-to-point architecture, which came with its set of obstacles. Whenever internal systems faced technical glitches or interruptions, it frequently triggered a flood of events, leading to delays and inefficiencies in resolving the issues. Moreover, the existing middleware solution required specialized knowledge for debugging and troubleshooting, hampering productivity and collaboration within the organization.
In the following sections, we will examine the specific challenges and benefits that Rapid7 experienced as a result of implementing an event-driven architecture for their order fulfillment process.
Challenges
Business Challenges
Rapid7 had to find a way to securely expose the order data to other systems as needed. The existing architecture’s reliance on an external middleware to access Salesforce introduced unnecessary complexities and potential failure points. The external middleware, originally intended to facilitate Salesforce access, created additional layers and dependencies, increasing the overall architectural complexity. This complexity stemmed from the need to manage and maintain the external middleware, as well as the potential for communication breakdowns or failures between the middleware and Salesforce, which could compromise reliability.
Technical Challenges
One of the challenges Rapid7 faced was the need to integrate near real-time Salesforce events while ensuring no data was lost. This crucial step ensured that the new architecture could seamlessly capture and process all relevant order data.
Rapid7 also had to ensure that the new solution could be implemented quickly, while laying the foundation for a scalable, robust, and secure architecture that could be built upon in the future. The existing solution required specialized knowledge to update. Integrating new internal and external systems also required extensive knowledge and code changes.
Finally, Rapid7 faced the challenge of improving near real-time error visibility and management for its internal support teams. The team required the ability to replay events that had been missed during an error situation to avoid data loss, a crucial aspect of the transition. It would enable the company to more effectively identify and resolve issues that arose during the order fulfillment process.
Solution Overview
Figure 1 — Solution Design
Solution Flow
- Capturing Order Updates: When a customer order is entered into the Salesforce system, the Event Relay integration relays this event to Amazon EventBridge. This allows Rapid7 to capture all changes to orders and distribute those updates to current and future systems that need to be aware of order status changes.
- Filtering and Processing Order Updates: The order updates received in EventBridge are then filtered, processed, and sent to external processor systems to create sales order and fulfillment records. This ensures that only the necessary information is forwarded to the downstream systems, optimizing the overall efficiency of the process. Dead letter queues, implemented throughout the process, capture problematic updates, enabling efficient error handling, reprocessing, alerts, and timely resolution for smooth system operation and data integrity.
- Receiving Licensing Information: The external processor systems, upon completing the order processing, notifies the EventBridge with the necessary licensing information. This feedback loop ensures that the Rapid7 platform has the required details to enable the requested features.
- Updating the Rapid7 Platform: The Rapid7 platform system is notified of the new license changes and updates the platform accordingly, enabling the requested features for the customer’s order.
- Communicating Successful Execution: Once the order fulfillment process is complete, the fulfillment processor creates a success update message and publishes it back to EventBridge, ensuring that the entire process is documented and can be tracked.
- Closing the Loop: The final step involves sending a completed record back to Salesforce using the API destination target, effectively closing the loop and completing the order fulfillment process.
Solution Components
- Amazon EventBridge: Rapid7 used EventBridge to build event-driven applications at scale. This allowed them to ingest and process events from a variety of sources, including their internal systems and external applications.
- Amazon Simple Queue Service (Amazon SQS): Rapid7 used Amazon SQS to decouple and scale serverless applications. This allowed them to manage and process messages between different components of their fulfillment process.
- AWS Step Functions: Using step functions allowed Rapid7 to have an ordered flow of execution, with some minor pre-processing of data prior to sending to external systems. The Step Functions orchestrated the flow of events originating from the event bus, validating the payload’s integrity. While invalid messages were routed to a dead-letter queue for further analysis, valid messages were seamlessly forwarded to external processing systems for subsequent handling.
- AWS Lambda: Rapid7 used Lambda to run their code without provisioning or managing servers. This allowed them to create and manage the functions that processed events and messages in their fulfillment process.
- Amazon Simple Notification Service (Amazon SNS): Rapid7 used Amazon SNS to fan out messages to multiple recipients for parallel processing. This allowed them to distribute messages to various systems and applications that had pre-existing integrations with SNS, enabling them to take appropriate actions based on the fulfillment process.
Benefits
Business Benefits
The journey of Rapid7’s order fulfillment transformation has been a testament to the power of event-driven architecture. By laying the foundation of a scalable, robust, and secure architecture, the team has paved the way for the continuous enhancement and expansion of their processes. The improved operational visibility and management results in a better experience for Rapid7’s customers.
The enhanced error visibility and management also benefits Rapid7’s support teams, allowing them to focus on resolving issues rather than sifting through logs, leading to a better customer experience. The centralized and streamlined logging and monitoring capabilities of the event-driven architecture makes it easier for the support team to quickly identify and address problems during the order fulfillment process.
Importantly, the foundation established with this event-driven architecture has set the stage for future expansion, opening the door to exciting possibilities such as customer onboarding events, product usage events, partner integrations, and go-to-market events.
Technical Benefits
The ability to scale up or down quickly and pivot swiftly without significant investment has given Rapid7 the agility to respond to market demands. The loose coupling between publishers, brokers, and consumers allows for more independent development, enabling the team to make changes and add new components or services without affecting the overall system.
Additionally, the new event-driven architecture offers replay-ability and recovery features that were not present in the previous approach. For example – If there are transient errors, such as connectivity issues or errors in the event relay configurations, the Event Relay connector between Salesforce and AWS would resume automatically after the connectivity issues are resolved or the configurations are fixed. This ensures that no events are lost, and the order fulfillment process can continue seamlessly, even in the face of temporary disruptions.
By working closely with the AWS team, Rapid7 significantly reduced the complexity and potential points of failure in their order fulfillment process. They overcame the complications of point-to-point architecture, streamlining the solution and improving reliability and resilience.
Embracing the power of event-driven architecture, Rapid7 has transformed its order fulfillment capabilities, positioning the organization for continued growth and success in the dynamic business landscape.
Conclusion
Rapid7’s journey to implement an event-driven architecture for their order fulfillment process has been a resounding success.
By employing an event-driven architecture on AWS, Rapid7 achieved their objectives of improving the fulfillment process, enabling flexibility, and increasing efficiency. The event-driven architecture offers greater visibility into the fulfillment process, enabling the team to quickly identify and rectify issues. Additionally, it provides a highly adaptable and scalable solution that seamlessly integrates with other systems and applications, ensuring a future-proof and flexible environment.
AWS played a role by helping navigate the implementation process. They provided architecture guidance and shared best practices, allowing Rapid7 to make informed decisions and avoid potential pitfalls.
Looking ahead, the foundation established with this event-driven architecture has set the stage for future expansion and growth. Rapid7 can now explore exciting possibilities, such as integrating customer onboarding events, product usage events, partner integrations, and go-to-market events, further strengthening their position in the dynamic business landscape. By embracing the power of event-driven architecture and working closely with the AWS team, Rapid7 has demonstrated its commitment to innovation and its ability to adapt to the changing needs of its customers and the market.
.
.
Rapid7 — AWS Partner Spotlight
Rapid7 is a leading provider of security data and analytics solutions that enable organizations to implement an active, analytics-driven approach to cyber security. Our solutions empower organizations to prevent attacks by providing visibility into vulnerabilities and to rapidly detect compromises, respond to breaches, and correct the underlying causes of attacks.