(2015) Building Fastly's Initial Support, Parts 1-4

Note: What started as a quick blog post in 2015 about support quickly ballooned into a multi-part novella. This is a consolidated post of the full content, relatively untouched.

Part 1: Our Standards

One of Fastly's greatest product advantages is our support infrastructure. We get compliments and positive feedback nearly every day through tickets, tweets, and old fashioned conversation. When we run case study interviews with customers, support is a common factor in why people choose Fastly over other providers. This is something we take pride in, and something we love to brag about.

This culture of support and service isn't an accident. Our support processes span the entire company, and are baked into every aspect of the way we do business. We worked hard to build our support infrastructure (our processes are so broadly encompassing and involved that calling our support an "infrastructure" is the only term that feels right) to scale without sacrificing quality, timeliness, or the soul that got Fastly where it is today.

In the time I've been at Fastly, we've had enough customers and friends ask us about how we do support, and what's going on under the hood that it seems appropriate to give a high level overview of how we build, what we've learned, and what other teams can borrow/steal from our setup.

The Context

When you look at Fastly's support, you have to start with looking at the context surrounding the company.

Infrastructure vendors historically have bad support experiences, unless you're spending something huge: think budgets in the tens of millions. Even then, support is viewed as a cost center -- the vendor's goal with support is for you to go away. If you going away is the result of them solving a problem, that's a plus. But the goal is to make you go away, solution or not. If you don't believe me, talk to your head of ops.

Because we're a company founded by operations engineers, for operations engineers, having par-for-the-course (ie: bad) support wasn't acceptable. Fastly's founding team set out to create the support experience they always wanted from a vendor.

The catch is: every startup begins with this goal. I've never met a business owner who feels like their company should have bad, or even mediocre, support. They all claim they want the best. That said, we've all seen cases where companies grow, become successful, and their support inevitably goes down the tubes. That pursuit of "the best" has been sacrificed, either intentionally or as an unfortunate side effect of other decisions.

I can't comment on other companies and their circumstances, but I can go into detail about the culture and processes that Fastly has in place to prevent this.

The Early Days of Yore

In the earliest days of Fastly, the team made some very crucial decisions about how they would approach support as a company.

From the outset, support was considered a profit center, rather than a cost of doing business. Fastly theorized that great support would generate more business than any sales force could in the earliest days, and approached setting up support with that in mind. It turns out, that theory was right, and customers flocked.

In the operations world, reputation is everything. Fastly already had respect in the community through the early team's technical reputation, but their support reputation became world renowned in quick fashion.

So, what actual decisions were made to set the foundation?

In the early days of Fastly, everyone joined in on support. Engineering, Ops, Neteng: they all took tickets they could answer, helped as quickly as possible, and communicated as honestly as they could.

Everyone shouldered the support burden, and took pride in delivering a great experience. The experience was so valued that people even "competed" to take tickets first. Great support was baked into the culture. Fastly's early employees had all run into horrible vendor support in their previous jobs, and were determined to do the opposite in their time at Fastly.

The people who shipped product answered the tickets about their features, wrote docs as needed, and made sure customers got the full benefit of Fastly. When new customers came in, everyone pulled together to make sure the migration went smoothly. It wasn't farmed out to a specific individual, the customer wasn't just left to their own devices. Infrastructure migration often needs extra help to make the process go well, so everyone chipped in. The customer's success was viewed as Fastly's success.

A buzzphrase in the startup community these days is "do things that don't scale." Fastly did this with support. It's not scalable to have your entire team onboard single customers. It's not scalable to have your CEO assist in real time over IRC. It's not scalable to give your everything to make sure the customer has a great experience.

The catch is: during those same early days, we did start scaling, and we did so very successfully. We started building the tools, resources, and everything else necessary to make sure this quality could be extended indefinitely into the life of Fastly.

It was said earlier: Fastly is a company by operations engineers, for operations engineers. Ops scales applications. Ops plans capacity. Fastly scaled and planned for capacity with support in this same way.

As an aside, many support teams are avoiding the concept of "scaling support" outright, as "scaling support" often implies cutting corners and removing the human touch that makes so many organizations successful. This should more accurately be considered "scaling support poorly." When Fastly scales support, we make the conscious effort to preserve the most essential facing aspects. If those aren't preserved, we view the scaling efforts as a failed (but valuable) test and roll back.

---

The next part of this piece will go into the decisions we made early on as an organization, the lessons we learned in the process, and what we've done to expand our quality of support and customer service. Follow me on twitter to hear when updates are live!

Part 2: Making Experiences

What customers encounter when evaluating Fastly, and as they onboard, isn't arbitrary. It's an experience that we've intentionally crafted. This piece will goes into what customers experience, and what we have in place to make sure this happens every time, for every customer.

The Customer Process

Before we go too deep into our support scaling process, lets take a quick look at what happens when a person or company signs up for Fastly. Inevitably, this shapes the support need and the service we provide, so it's important to keep this in mind.

Most customers come to us with a scaling or ops need. They may be looking for a new CDN vendor, they may have a difficult problem to solve such as scaling a service API, or they may just be growing exponentially. In any case, there's an obvious operational pain point that we can address.

We intentionally provide a frictionless signup process so customers can begin configuring and testing without needing to talk to a human first. We're ready to assist, and we make sure that new customers feel free to reach out, but we don't force ourselves into the picture unnecessarily.

This is a core value of Fastly's support: we give you as much (or as little) assistance as you want, exactly when you determine.

Inevitably, customers do reach out with questions during the setup process. This is infrastructure: communication is to be expected and encouraged. Either over some email or phone exchanges we discuss fine tuning the system for the customer's needs. We're intentionally flexible as a service: you know your application and needs better than we ever will. But that doesn't mean that there aren't onboarding questions.

Some customers can configure their service on their own, and move traffic over quickly and easily. We've put a great deal of work into our documentation and application design to enable self service. But, there are always cases where our teams need to assist. In that case, we may spin up an IRC room for our team and yours too coordinate the switchover, or debug any setup issues in real time. Sometimes chatting for 15 minutes can save days of frustrating email exchange. We'd rather get to the bottom of things right away.

At this point some portion of a customer's traffic has been moved over. There may be a slow ramp over time. We may be serving a niche portion of a customer's traffic. We may get the 100% of a customer's public traffic in one day. In any case, some traffic is now flowing through the customer network.

In many cases, this is where the interest and support ends. With Fastly, it's only the beginning.

Because we're a CDN, we need a connection to your origin server. This gives us the ability to see service issues or error spikes almost immediately. Our teams can proactively reach out in the event of a service disruption. We then work with you to as needed to adjust your CDN or troubleshoot issues.

This ability to proactively assist is possibly the most impressive post of our support. If you're planning support and you have one takeaway from this piece, it's to find a problem your users have that you can predict, and proactively reach out when you see issues arise. A fast response to a customer's ticket is amazing, but if you can spot a problem before it starts that's even better.

At some point a customer may want to put more traffic on to Fastly, fine tune their current traffic for a higher hit ratio, build in automated purging or set up detailed logging from the edge. This is a more traditional, consultative support. We view ourselves as scalability partners for our customers, and this is an extremely rewarding part of the job. We push the limits of what can be offloaded, and it's fun to work with customers from an architectural standpoint to help hit their goals.

This has basically been the support process for every single Fastly customer from day one, with very few exceptions. This experience is what we work on scaling and improving. We've absolutely made some changes to how we prepare and carry this process out on our end, but there have not been major changes to what aim for when providing support. You should be able to speak with an early Fastly customer and a recently ramped Fastly customer, regardless of size large or small, and those two customers should describe roughly the same experience. That's not by accident.

The Support Process

So what's happening under the hood in the above section? What steps and processes do we have in place to make customer setup and traffic ramping go smoothly?

Pull up a chair and we'll go right into the thick of things.

Our process starts when the customer contacts Fastly to investigate the product, or shortly after the customer begins configuring a service after signup. We then discuss their end goals over email, phone, IRC, or in person. More often than not, our documentation base can help them get set up with most configurations. But, in the off case that our documentation doesn't address everything, we work directly with the customer over the course of a few days or weeks to build up a configuration that does what they need.

This is configuration is usually done in our web application, but occasionally we dig in and write some VCL (Varnish Configuration Language, the scripting language that powers our caching system) to get things done. We may help with setting up logging and analytics, but those are easy to self-provision.

Now we're ready to start testing traffic with the customer. In prep for this we work with our ops and neteng teams to forecast traffic. As we're now serving a consistent 1.8 million requests per second worldwide, individual sites and customers don't cause the network delta they once did, but we still loop in everyone. When it's time for a customer to finally flip the switch, we have a group of folks standing by from ops, neteng, product engineering, and support.

Of course, this process always doesn't happen exactly as I describe here. The process is as unique as their own application, so we help iteratively as needed. Every few weeks, customers may come back and file a ticket or two, or bump us in IRC, but the bulk of the work is up front.

The actual coordination of these processes is jointly run by the customer's account manager and our support team. Our account managers typically will get the relevant information regarding a turn up, and plan with our support and engineering teams for a successful traffic switchover. The main takeaway is that the entire company contributes to the customer success. It's the majority of what our support team does on a daily basis, but every team at Fastly is partially involved with and responsible for customer success.

Next Time

The next piece in this series will dive down into the structure of Fastly's support team, and how the team interacts with the rest of the company. This has been the most requested topic so far, so I'm excited to publish it. All things equal, this team and cross-company interaction is possibly the most unique aspect of our support architecture. Follow me on twitter to hear when updates are live

Part 3: Tactics, Tools, Teams

Fastly has made some key decisions about how to operate and structure our support team in a way that allows us to deliver quality at scale. Some of the lessons we learned are more specific to issues we encountered as we grew, but most can be applied to any support or customer-facing team.

Don't repeat yourself

The first support pain point Fastly ran into was answering repeated questions, as every company might experience. There are two major tactics that can be used in some combination for solving this problem: documentation and canned scripts in the ticketing system (or a text expander tool). Adding people to the team is a solution, too, but that isn’t possible for many companies, as it's by far the most expensive and least efficient way to address the problem.

In true ops fashion, Fastly embraced DRY (Don't Repeat Yourself) in full force. If a question was asked twice, someone built the answer into a document on docs.fastly.com. The team didn't use canned scripts or macros at all. The nature of our questions didn't make reusable scripts practical -- even today there are not a lot of canned scripts used due to the nature of our support requests.

This presented a few issues. Sometimes the docs were unorganized, as nearly every employee contributed, and sometimes the docs overlapped. At the time, there was no one owning or coordinating documentation efforts.

But, it was a net positive, as it tapped into a support truism: many people would rather not contact support at all. If you can make it easier for your customers to self service, you're making it easier for everyone. It also makes it possible for customers to solve problems themselves, which makes them feel awesome. And who doesn't like feeling awesome?

It's important to note that when a customer contacts support, they are taking a lot of effort to seek assistance and admit they need outside advice. For some people, that's a normal process. But for others, it can be a huge ego hit. If you can make self-service easier, either through product design or documentation, your customers will have a better experience with your product, which directly shapes their support experience.

The right tools for the job

Another friction area Fastly encountered early on was the need to communicate through more than just email or support tickets. Since everyone in the company (and most of our early customer base) was already in IRC, it made sense to set up an IRC room for customer communication. The room #fastly on Freenode was born. It's still there. You can ping me in there right now, most likely (I'm @aspires).

IRC is an amazing way to communicate with customers in real time to either debug problems or talk shop. You'll get some of your best feature requests and product insights from the users who take the time to chat with you. Also, Fastly has reached a critical mass where customers are helping other customers. That's amazing, but it's not unique to us. It’s happened at every company I've seen with a maintained IRC room.

IRC helps us scale in another way. When we're working closely with customers, we can set up rooms for the customer quickly, and get down to work without a lot of overhead. Customer onboarding for major infrastructure changes needs to be faster and smoother than email, but not as time-consuming and cumbersome as a conference call. IRC offered an ideal, asynchronous communication tool that only chat can provide.

We later found that when customers helped other customers in IRC, a lot of shared knowledge wasn't getting logged in a reusable way. So, we've recently launched a Discourse-powered Community forum to help keep the interesting use cases and shared knowledge in an indexed location.

We also have a lot of unique workflow-improving tools for our specific circumstances. Some are built right into our ticketing system, and others live in a central location outside of the ticketing system. Nearly every support team has them, so this isn't a huge surprise. But I do recommend building these as early as possible. A great support toolset, combined with comprehensive documentation, will scale your support organization more than you think. It's also fun to blow off some steam hacking on an internal tool that will get real use.

Another tool we use heavily at Fastly is our pager system. That's right, support carries a pager. It's a follow-the-sun rotation with our London office, which eliminates the risk of being woken up. It originally started as a way to track who would field tickets over the weekends, but we've applied the rotation to different tasks. For example, we have Premium Support customers with an emergency trigger email, which pages our support team. It also means that if there's an issue that needs support assistance, our engineering teams have the ability to page us.

A dedicated department

I said earlier that Fastly started out with engineering answering tickets directly. It was a great process that helped fine-tune our initial product set in the best way possible. Fastly was able to make this scale further than most. With the techniques and tools previously outlined, Fastly engineering was able to do their own support for almost two years without a dedicated support organization.

But, as ticket load increased, and as teams across Fastly began to specialize, there came a time to build a real support team. Fastly has structured support as part of a larger organization called Customer Engineering, responsible for the entire customer onboarding and lifecycle process. This includes sales engineering, support, documentation, and professional services. Some teams call this "Customer Success," but the result is the same -- support is factored into the entire business process, rather than operating as an outside team.

Fastly officially started the customer engineering team in mid-2013 with support being the primary responsibility, but the team took on sales engineering work that was previously done by the founders and a few early employees. At the time of writing, Fastly's customer engineering team has nearly 20 people divided among our SF, London, and NYC offices.

Team structure

That leads into team structure — not just how our support organization is structured, but how we interact with the rest of the company.

Customer engineering is a first-class organization at Fastly that actively works with all teams to ensure a high-quality customer experience. We have the resources we need to grow and make decisions for the good of the customer, and because we're structured to address the entire customer cycle, we can have real impact on customer happiness and product perception.

The customer engineering organization is also included in the product development process. We don't dictate what's happening, but we do get copied on projects as they progress. Because we're looped in regularly, we're set up for success; things don't ship as a surprise, and when things do ship, we've done the necessary work to be ready for go-live. Because of this, we're better prepared to manage inevitable bug reports and customer issues after ship.

This is critical for good support and happy customers -- simply looping your support team in on the development process early will make your ships significantly more successful than pushing features without communication.

Fastly's process for solving tickets with other departments is also unique. Typically, support organizations operate on their own island, where they either cannot escalate a ticket to product engineering for assistance (the common startup scenario), or where they escalate a ticket to a magical "next level," never to see it again (the common big company scenario). Neither of these is ideal. You're either building a system where customers run into barriers and red tape, or a system where support pushes the buck to other people.

Fastly intentionally structures its support workflow in the following way for escalating and interacting with other teams:

1. New ticket comes in.
2. Support claims the ticket and starts investigating.
3. Support comes to a point where another team needs to weigh in.
4. Support assigns the ticket to the team's group in the ticketing system (support is still copied on the ticket, as well).
5. The assigned team is now responsible for providing an answer, update, or response on to the ticket thread.
6. Someone from the assigned team weighs in, possibly responding to the customer directly.
7. Support takes the ticket back, and continues solving the issue. At this point, if necessary, go to #4 and repeat.
8. Support resolves the issue with the customer.
9. Support starts the documentation process as needed.

It's a little more involved than other organizations, but the important steps here are #5 and #7. When support assigns a ticket to another group, such as ops, finance, legal, neteng, etc., that team works to get an update as soon as they reasonably can. Once that team weighs in, it’s support's job to take that, relay it in a meaningful way to the customer, ultimately resolve the issue, and create documentation (if appropriate).

Our technical teams have been experimenting with an "on call" rotation for interfacing with other teams on pressing issues. Fastly just happened to use pager shifts like this to structure other things, so it made sense to build support (and other teams) into the shift process for communications. You may need a different system, but the main point is this: great support teams need to have a codified way to work with the rest of the company in a way that doesn't distract or derail daily work.

If support isn't included in the final process of shipping products, and if your support organization doesn't have a respected way to pull in other teams to help solve customer problems, you're hindering your efforts to build and scale the quality of support you offer. The people who end up suffering are your customers and users.

This piece definitely got more into the trenches of planning, team management, and the actual process of growing support. So if you have any follow-up questions, let me know at @austinspires on Twitter.

There’s one post more left in this four-part series, so stay tuned.

Part 4: Scale

This article dives into our support team's training processes and explains how our lessons learned can help your support team grow.

Training

Many customers come to Fastly’s support seeking advice on Varnish, an open source software that’s at the core of our CDN services. Because of this, our team has to be experts on all things Varnish. We also have an incredibly technical product and customer base, so we have to be equally knowledgeable. Even for team members with prior CDN or Varnish experience, Fastly’s technology requires training and continuing education on a regular basis to maintain meaningful support interactions. If you're running a support team, large or small, proper new hire training and continuing education are essential to keeping your quality and team morale up.

Because of this, Fastly needed to embrace having a comprehensive training and onboarding process for our new and existing support team members in the early days. Our current training programs are not formally completed (yet), but we have everything from advanced technical talks with our subject matter experts to informal computer science classes covering the absolute basics to side-by-side pairing sessions. Fastly has reached the point where nearly any new hire can be brought up to full productivity within a few weeks.

The training I mentioned above focuses on just the technical aspects of the job, not the softer skills that make up support interactions. We have training processes for that, too. Support is a skill, and it takes practice. From spotting related problems from specific wording in a ticket to being able to effectively convey appropriate emotions in writing, language skills are crucial for support.

Timing and tone

With proper technical and tone training in mind, we’ve discovered something surprising as we’ve scaled: using a relatable tone and human voice are extremely important to proper support, but fast initial response and correctness have a much stronger correlation with customer satisfaction in a support interaction.

In practice, there’s a sliding scale effect. If your team immediately responds to customer tickets and solves customer problems, tone is less crucial because you’ve promptly mitigated risks. But if your team’s time to first response is long (24 hours or more), having an approachable and human tone is perhaps the only thing that can improve your relationship with the customer. In the worst possible scenario — taking an extremely long time to respond — you must have both great technical answers and an impeccable tone. This is the hardest response to achieve reliably. Given the option, encouraging and measuring a fast response time is the course of action that benefits most companies today.

Takeaways

It’s hard to scale an effective support organization in the long term. You have to care about support and the customer experience more than you ever thought possible. You need to build tools and processes around support like you would your flagship product. Most importantly, you have to be willing to change your assumptions as your customer base scales and changes. The needs of your early customers will be very different than the needs of your later customers, but the care and respect you show them will be universal.

There’s not a lot of resources on scaling support available, but there is a great deal of writing on scaling operations, capacity planning, and structuring technical teams. Believe it or not, these may be the best resources for your company’s support needs. There are obviously some areas that don’t overlap; you should never treat your support team like interchangeable machines, for example. However, the process of planning based on business growth is a great mindset to use when architecting your support.

Most importantly, there’s a lot of non-advice about support out there. Telling your support team to “care more” or “engineer happiness” or “close more tickets” isn’t support leadership, as it’s not clear or actionable. Support is a process, like any other facet of building a company. You have to work at it, define your customer process effectively, and be pragmatic about what systems you need to scale. It’s not glamorous, but the sooner you can address your customer’s needs, the faster you can build a support organization that’s actually meaningful and helpful.

Scaling support can be difficult, but it’s not impossible. I shared our history at Fastly to give some insight into what it takes to build a great support infrastructure at your company. Fastly’s process will likely not fit your specific business needs, but hopefully our lessons learned can be of some benefit to you.