Security Summary Dashboard

For official documents, click here: https://techdocs.akamai.com/etp/docs/security-summary-dashboard

The problem

In my role as a senior product manager at Akamai, one of the interesting conversations I used to have with customers was about proof of value. Whether it was a new prospect trying out our product during the trial period or an existing customer who had used the product for a long time, the question of how to show value came up in almost every conversation.

When I joined Akamai, the lack of a robust proof of value report was one of the first things I noticed, especially since proof of value reports are table stakes in many of the enterprise products I had owned previously. The customer facing teams relied on a manual process to create a value confirmation report. This report was generated using scripts and took quite a bit of time to finalize, because of the significant data cleanup needed after the report was generated.

Discovery

Before writing the PRD, I wanted to understand a few things:

What exactly would a proof of value or value confirmation report look like?
Who would the report consumers be?
How would we deliver this report?
What return was I expecting from this report?

To get answers to these questions, I did the following:

Reached out to the account teams of strategic customers for their input on the value of such a report, and on the likelihood of their customers partnering with us on the design.
Held focus group discussions with some of our trusted customers to understand how they consume our reporting and how they demonstrate the value of our product within their organization, especially at the time of renewals.
Reviewed the existing report and walked through some of the customer specific reports with the field teams (customer success, professional services, and pre-sales) to understand who the various users of this report would be and what they would like to see in it.
Researched competitive solutions to see what they provide in similar reports, how they deliver them, and how valuable their customers find them.
Held technical discussions with the architects and engineering leaders to identify the various components required to generate such a report.
Built a business case showing how such a report could improve our win rate and increase stickiness with existing customers.

Scoping decisions

Based on all this information, I drafted the initial PRD. I chose enterprise customers over service providers for the first release. An average enterprise customer had between 6,000 and 10,000 users, with the largest at around 50,000, while service providers ran into the millions. The lower volume of data made the report practical to build, and the value of such a report was also higher for an enterprise customer.

I also had to decide what the dashboard would leave out. One of the main roles of a security operations team is continuous monitoring and incident response. That use case relies on log analysis through an integrated SIEM, so it was not intended to be solved by this dashboard.

What we built

My primary objective was to improve the win rate of our PoCs, since the post-trial phase is where a value confirmation report is most valuable. The long term goal was to provide a proactive report so that customers understand the return on investment they are getting from our product.

The report had three main users:

Security operations professionals, whose main goal is to keep the customer environment secure. To achieve this, they need to understand the threats they face along with the steps to mitigate them.
Network and IT administrators, whose main goal is to apply organizational policies to managed devices and employee user accounts. To ensure the policies are applied consistently, they need to see the success rates of those policies. Where policies failed or were subverted, they want to understand the reasons behind the failures so that they can undertake remediation.
Executives, typically CSOs, though other members of the C-suite were sometimes involved. Their main goal is to ensure they are getting a return on their investment. To justify purchasing our solution, replacing a competitive solution with ours, or renewing, they need to understand how our solution protects their users and which best practices would help them get more out of it.

The MVP was organized into three sections:

Threat Summary: Intended mainly for executives, with the security operations team as a secondary user. It covered the current risk score, the trending risk score, recommendations on improving the risk score, threats blocked, and the percentage of malicious activity.
Threat Analysis: Intended for security operations professionals to deep dive into the threats seen in the network over time. It covered blocked and active threats, their severity, details of the top threats including mitigation, and the devices or users at risk.
Network Analysis: Intended for network and IT administrators. It covered the number of devices or users in the network, devices or users not covered under the acceptable use policies (AUP), AUP violations including unauthorized applications, and traffic analysis over time.

The dashboard was delivered through the product console, with the option to export it as a PDF or sign up for weekly delivery. The PRD also defined dashboard loading times for the MVP and beyond, along with the internal data preparation needed to render the dashboard, especially for reports that required trending data.

For the MVP, success meant adoption among the customers who had the report enabled, positive feedback with a documented use case and value from at least 10 customers, improvements in win rate and renewal rate against previous benchmarks, and no P0 cases caused by the new dashboard. After GA, the focus shifted to engagement across the install base within 6 months, alongside win rate and renewal rate.

Earning the funding

Once the PRD was complete, I had to demonstrate the business value of the dashboard to the leadership team. The initial approval was only for discovery and analysis, not for full development.

Working with the UX designer, we created mockups of the dashboard. These mockups were reviewed with our design partners, and the process of collecting feedback and refining the design carried on iteratively. In parallel, I manually generated some of the reports and started presenting them at customer QBRs. The feedback from customers who saw these reports was positive and helped refine the solution even further. With all the feedback and a demo in hand, I presented to the leadership team, who then approved funding for the Security Summary Dashboard.

The hard parts

The biggest technical constraint was data aggregation. As a policy, we retained customer data for a limited number of days, and this feature pushed that limit to 12 months. Engineering only wanted to show 90 days of data. With 90 days, what we showed would barely be a fad, let alone a trend. After multiple discussions, we agreed on 12 months for certain data sets. We made tradeoffs to make this work. For example, the risk score was calculated once a month and only 12 values were stored, instead of a value for each day.

Another challenge was bringing telemetry and insights from other teams into the dashboard. This required negotiating data-sharing agreements across teams, and it proved harder than much of the engineering work.

Outcomes

Working with the engineering team on a cross functional effort involving UI, data engineering, security, and network teams, we launched the MVP to an initial set of customers and enabled it by default for all new PoC accounts. The win rate improved significantly against the previous benchmark. The renewal rate hit 100% for the initial set of customers, although this may be due to bias in how those customers were selected. The feedback from the field and from customers, including customer zero, Akamai itself, was overwhelmingly positive. Funding for the full GA launch was approved, and the dashboard reached general availability shortly after I moved on.

The longer term roadmap included a Global Threat Landscape report, which used Akamai's threat intelligence across products and customers to show the top threats in each geography and sector, so customers could compare how they fared against their peers. It also included an analysis of the shadow IT and shadow AI tools being used without IT approval. Both have since launched. Other themes on the roadmap included a richer risk score drawing on telemetry from all connected Akamai products, more context aware recommendations, and faster dashboard loading.