At Centercode we believe in technology's ability to make everyone's life better. In the Delta Huddle Podcast, we bring together industry experts and visionaries to share their insights about building products and bringing them to market.
We're excited to announce the next episode of the Delta Huddle podcast! Joining Centercode’s VP of Marketing Chris Rader to discuss the past, present, and future of User Testing is Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda, and Mario Sancho, Chief Delivery Officer at Centercode.
This episode of the Delta Huddle Podcast is really special, for a lot of reasons, mainly because it delves not only into the heart of user testing and what drives it forward, but also its history and where it came from. What we think of today as a “traditional beta test” or a “standard product test” didn’t start out that way. It took years of trial and error and learning about what makes testing great to get to where we are today with delta testing.
In this episode:
- Testing during the dot com bubble
- Why fewer resources results in an increased need for beta programs
- Why delta outperforms beta as a testing methodology
- The value of a good beta program
- How to convince a company to start a beta program
- Perceived reason for testing vs. actual value of testing
- How user testing helps break down silos in an organization
About Our Guests:
Sharon Rylander’s many accolades and accomplishments include winning a technical Emmy for her work at Sling, collaborating with Pixar Animators to bring miniature robots to life, and showing state of the art VR technology to the late Stephen Hawking. In the past 15 years, she’s worked exclusively with startups to help build out teams for UX, beta testing, customer support, and more, regularly intersecting with the fields of analytics, marketing, and PR. Her current role at Square Panda focuses on foundational literacy games for young children.
With nearly 20 years under his belt at Centercode, Mario Sancho brings a wealth of experience and expertise as the Chief Delivery Officer. His extensive background in beta test management, including roles in support, training, and sales, has been instrumental in ensuring that Centercode’s customers continue to receive the maximum value from all our services.
Follow us and listen to the Delta Huddle Podcast on the go:
- Apple Podcasts: https://apple.co/3V5x74Q
- Google Podcasts: https://bit.ly/3Wq0HTT
- Spotify: https://spoti.fi/3PNYCiJ
- Audible: https://adbl.co/3Pvw90V
Episode Transcript:
00:00:00
Stefan Stenroos, Technical Trainer at Centercode
This is the Delta Huddle podcast.
00:00:08
Stefan Stenroos, Technical Trainer at Centercode
Hello, I'm Stefan Stenroos and welcome to the Delta Huddle Podcast. Today's episode is really special for a lot of reasons, mainly because it delves not only into the heart of user testing and what drives it forward, but also its history and where it came from. What we think of today as a traditional beta test or a standard user test didn't start out that way.
00:00:29
Stefan Stenroos, Technical Trainer at Centercode
It took years of trial and error and learning about what makes testing great. To get to where we are today with Delta Testing. Joining Centercode’s VP of Marketing Chris Rader to discuss the past, present and future of user testing is Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda and Mario Sancho, Chief Delivery Officer at Centercode.
00:00:51
Stefan Stenroos, Technical Trainer at Centercode
Sharon's many accolades and accomplishments include winning a technical Emmy for her work at Sling, collaborating with Pixar animators to bring miniature robots to life, and showing state of the art VR technology to the late Stephen Hawking. In the past 15 years, she's worked exclusively with startups to help build out teams for UX, beta testing, customer support, and more. Regularly intersecting with the fields of analytics, marketing and PR.
00:01:17
Stefan Stenroos, Technical Trainer at Centercode
Her current role at Square Panda focuses on foundational literacy games for young children. And with nearly 20 years of experience under his belt at Centercode, Mario brings a wealth of experience and expertise as the chief delivery officer. His extensive background in beta test management, including roles in support, training and sales, has been instrumental in ensuring that Centercode customers continue to receive the maximum value from all of our services.
00:01:41
Stefan Stenroos, Technical Trainer at Centercode
This episode was a joy to make, and the insights shared by Chris, Sharon and Mario were absolutely invaluable. There's decades of wisdom to be found in this episode, and I hope you enjoy.
00:01:52
Chris Rader, VP of Marketing at Centercode
And I understand you guys have a pretty extensive background. So, Sharon, first, I'd love to hear how you met Mario.
00:01:59
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah! So, I worked at the time at a company much like Centercode, except we predated you guys called BetaSphere, and we were in a massive time of growth trying to hire as many program managers as possible. We were basically doing outsourced beta program management. We didn't have a SaaS model yet, so it was strictly just the services and Mario's resume came across our desk and it was shy on some of our required skills.
00:02:28
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
But I was the person during the phone screens and he kinda just barely passed. And you lived in SoCal. And so it was I felt the weight of this decision about whether or not to even fly him up for an interview because on paper he looked kind of weak. But I talked to him and, you know, there was something there.
00:02:49
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And I took a chance and we brought him up and he ended up being one of our best employees. It was really amazing.
00:02:55
Mario Sancho, Chief Delivery Officer at Centercode
You’re so kind!
00:03:01
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Of the grief I gave you after you.
00:03:03
Chris Rader, VP of Marketing at Centercode
Met through paper.
00:03:05
Mario Sancho, Chief Delivery Officer at Centercode
Yeah. The other side of that coin. And the reality is she absolutely took a chance on me there. Anybody that's worked with me over the years with beta… if you don't like me, she's the one to blame. And if you like me she's the one to blame as well. But at the end of the day, I was actually just out of the military at that point.
00:03:27
Mario Sancho, Chief Delivery Officer at Centercode
I was in the Navy as a reactor operator, been out of leave a year or two, and I was working as a field engineer for a company called ADT at the time, doing testing and installation of their equipment, electronics, all the article surveillance, everybody at that point was doing the dot com. So all my friends were going to dot coms, and I wasn't necessarily interested in and going into tech, at least into the startup world.
00:03:53
Mario Sancho, Chief Delivery Officer at Centercode
But I had had an interesting experience at the time where I had done really well with the company I was at, got to my first review with the company and I was supposed to have been qualified in like six of their systems by that time. And they had a total of like, twelve, I had qualified and was certified and was being built out on all twelve of their systems as a field engineer.
00:04:17
Mario Sancho, Chief Delivery Officer at Centercode
So I thought I was just like, I'm going to get promoted, I'm going to the next step. And my manager said, You're doing a great job and I see no reason why you won't be promoted in the next 3 to 5 years. So that was when I said, Yeah, I want to do something different, that my effort can have a much closer result.
00:04:35
Mario Sancho, Chief Delivery Officer at Centercode
So let's go look at this tech and startup world thing. So, at that point mostly military background, and one one maybe two years of field engineer work, I started to put my resume out there and I got lucky enough to to talk to Sharon at the time. And she was actually very straightforward like, well, why, why do you think you can do this job?
00:04:56
Mario Sancho, Chief Delivery Officer at Centercode
And I'm like, Well, because I believe in myself and I am going to work harder and well, if I don't know it, I'll learn it. And I still remember her answer to this day, I believe you.
00:05:08
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
That’s funny! Sounds like me. Yeah, you were!
00:05:13
Mario Sancho, Chief Delivery Officer at Centercode
Yeah, I came in, met the team and, ah, say that the rest is history. And I got a job as what in essence today would call a Beta Manager. Yeah.
00:05:23
Chris Rader, VP of Marketing at Centercode
And Sharon you mentioned it was a time of, of growth and looking for someone to fill both the seat and specifically the beta program manager. What was going on that, that that growth was you know explain the growth if you could.
00:05:37
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah it was 2000. The 1999 - 2000 era. Remember that era? Yeah. When everything was exploding and there were just not enough bodies like, you know, literally you could go into Fry’s [Electronics] and see somebody who was homeless last week, you know, running the aisles looking for stuff for you because there were just not enough bodies to go around to do jobs at that time.
00:06:01
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
There's so much money flowing. And then, of course, 2001 hit and it all just cratered. And you know, went 180 there. But I, the less flippant answer is that these two guys who started this company Betasphere, they hit a nerve and I give them so much credit, they ran a beta program. It was successful.
00:06:25
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
They made a little money. They did another one. They kept growing for the first 16 quarters of their existence. They were growing and profitable on bootstrapped earnings, not a dollar taken from anybody, which is almost unheard of here in the Valley, you know. So it was impressive. And I think what they did, you know, it’s a niche area.
00:06:51
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Right. They hit a nerve. And I used to joke that we worked with everybody except Apple. But we were playing with all the large brands. It was startups and everybody in between. And so word got around and we were just growing rapidly.
00:07:09
Chris Rader, VP of Marketing at Centercode
Yeah. So yeah. So what about tech do you think was interested in specifically, I guess the beta management or the beta testing aspect of it.
00:07:21
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Sorry, what was the question?
00:07:22
Chris Rader, VP of Marketing at Centercode
So like, tech was looking for the solution, right? So like what do you think was happening specifically with tech? Like why do they need users? Why did they need testers?
00:07:31
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, now, at least at that time and it probably hasn't changed a ton. Is that... There's so much competition and so much money flowing around that companies could look at quality a bit more than sometimes they had in the past and that became a competitive advantage. There's a lot of stuff going around where people would buy technology and then consumers would get burned, right?
00:08:01
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And so especially if you've got a consumer, you know, consumer electronics, right, where it's expensive, really, really expensive. And it yeah. And what happens in a ton of companies and even to this day is beta becomes this checkbox. And historically that's kind of all it was is ,you know, you had your engineering and then you had your QA testing and then you had this checkbox of beta, and it was always an afterthought.
00:08:30
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And by the time you got to that point in the product development process, everybody's burnt out, they're just done and they're ready to get it out. There's a ton of pressure to get the product out to market and I know I'm not telling you anything that you don't see on a daily basis, right? There's just that pressure. And so it just gets done poorly.
00:08:53
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And plus it's tacked on to somebody's job description. Generally, the product manager. Right? And so they're doing a whole bunch of other stuff. And now they have to do this beta program, which it turns out can be very time intensive. And so it fails. They do all these rookie mistakes and it just doesn't… And then you don't get the benefit of what you could be getting.
00:09:16
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So these guys came along and actually said, you know what, your beta participation rates, that was a big thing at the time, beta participation rates, are maybe 30%. On a good day. We're going to guarantee at least 70%. And they did. And they did it through having a good methodology that was outsourced and basically took advantage of efficiencies across time.
00:09:41
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Because the other reality for a lot of companies is that they were only doing this checkbox maybe once every year or two because, you know, if even your, you know, Agile hadn't quite taken hold yet, so we didn't have that concept. So it was a you know, it was a longer term cycle. And so it didn't make sense for a company to hire a dedicated beta person unless they had out a whole bunch of hardware products.
00:10:05
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And then they could, you know, economy of scale, hire somebody. But that was more rare, right?
00:10:11
Mario Sancho, Chief Delivery Officer at Centercode
The other thing that I remember from that era, Sharon... a big portion of their value prop was what they would call the beta crunch, a resource crunch. And again, back to the era where we didn't have enough people to do anything. So this checkbox, whether it was a checkbox or not, or they tried to take it seriously, we… so I remember to this day the image this is your release date and this is the resources that you need right up until the release date.
00:10:36
Mario Sancho, Chief Delivery Officer at Centercode
Coming up they are going to be really high, and those resources at the same time trying to figure out how to launch product, pricing, distribution, product managers are up to here with stuff to do and you need to do this beta as one of the last steps along the way. So someone would take it as a checkbox and like, yeah, we did give it to someone and others would try to take it seriously.
00:10:57
Mario Sancho, Chief Delivery Officer at Centercode
And realize that they don't have the resources for it. So that was the big value proposition at the time that if you want to get something out of beta, look at Betasphere, cause we are your extended resources to get this service done in a quality manner and and and good high participation rates that you get something out of it.
00:11:15
Chris Rader, VP of Marketing at Centercode
Mhm. Yeah, it's a good take. Oh, How do you think that's for both of you? How do you think that's changed now? I know that was obviously, you know, 2000. There was an internet bubble. There's a lot of money. I don't think we're necessarily in that position right now in a lot of companies where money is very abundant.
00:11:33
Chris Rader, VP of Marketing at Centercode
I do feel like there's still not enough resources for what I'm seeing, that there's still few resources, but not as much money going around for that. So do you think that changes the idea or the concept of beta?
00:11:50
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
No, not at all. In fact, with less money, there's… it turns out, an increased need because you have even fewer people to get everything done. The thing that I've seen in my experience up close 100%, once a company gets a taste of a well-run beta program, they never turn back. They have to have it. It becomes a mandatory part of the equation because they see the ROI.
00:12:19
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
They get it. Until they experience it, oftentimes they don't get it, but that's what it takes. So especially when things get lean, they start relying even more heavily on, you know, outsourced, you know, tools and or services. So that's what I have seen. I think, in terms of changes in the industry, I mean, there's definitely been a lot of changes.
00:12:43
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
You know, now that, you know, everybody is doing some form of Agile or at least pretends to be, there's a lot more, you know, iterative releases out out there, you know, And that lends itself to the idea of delta testing, which I know you guys have coined, which is great! In the… you know like at Sling, we were basically doing that organically.
00:13:05
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
We just didn't know what to call it yet. So thank you.
00:13:07
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So I and what I mean by that is you could have a large scale beta program that is perfect for a new product going out to market for the first time. But once you're iterating on an existing product, that changes the equation a little bit in terms of best practices and also in terms of what you can get away with, you don't have to have so many people, etc.
00:13:38
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And so that's where Delta comes in and where's this ongoing process. And you're constantly iterating and it's just the next step after QA. Yeah.
00:13:47
Mario Sancho, Chief Delivery Officer at Centercode
The other part that you’ve seen and that I experienced here from then to now beta itself is better understood back then and the problem with beta fundamentally is that beta is both a stage and it's a process. So when you would try to say, oh yeah, we do beta and you'd always have to do a little, can you tell me about that?
00:14:10
Mario Sancho, Chief Delivery Officer at Centercode
And I'm like, Well, yeah, our tests are… it’s being QA’ed and our product’s in beta right now. Well who’s testing it? Well, our QA testers. Now obviously, that's not how we use it for beta and I’m like then you're not doing beta at all. So then that's now you can have a conversation that everybody's kind of coming from the same world when you're talking about beta, but it is truly a product that's out in the field.
00:14:34
Mario Sancho, Chief Delivery Officer at Centercode
That your testers or proxy for your testers, y’know nowadays we have dogfooding that would potentially be your customers… but it’s out in the real world somebody is either testing it as either proxy for your testers or your testers giving you feedback during a set period of time or all the time, if you will. That's where you can have a common conversation with just about anyone in any field now, about when it comes to beta.
00:14:58
Mario Sancho, Chief Delivery Officer at Centercode
But I think one of the biggest changes that I've seen since then and of course the methodologies, the experience that people have, the different products and what they can get from it, the value extracted from it for different teams, whether it's the engineering teams, the QA teams, the product teams, the UX/CX people, and even just straight marketing or promotion purposes are all kind of pulling from it today much more than they ever were back in the 2000s, which was mostly a quality play back then.
00:15:27
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yes. Yeah, yeah. Almost like an extended QA. And I agree 100% that the intersections with other departments is pretty high. The value there is pretty high. I just want to go back really quickly, though. So it sounds like Mario, I know you talked to a lot of companies. Is the nomenclature getting more standardized? Because I know this is one thing that has always driven us crazy, right?
00:15:48
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
The word beta, it gets used in so many different ways that it almost becomes useless. It sounds like… Are you experiencing a bit more standardization?
00:15:59
Mario Sancho, Chief Delivery Officer at Centercode
It is on the concept of the beta, but as you’re saying, we're now beyond beta. Centercode is beyond beta now. We’re in Delta, which is more in line with Agile. So if I want to have the starting conversation on beta that’s usually where I would do that. Well, we're… Fundamentally, we've worked in the past with beta tests and in that scenario I can have a very good quality conversation with clients that we're both talking about the same thing. It’s the next level where we're now introducing delta, this concept of continuous testing with better releases, it really starts to blow people's minds that you could do that like, Well, yeah, delta can absolutely...
00:16:38
Mario Sancho, Chief Delivery Officer at Centercode
It's well-timed for that. A well organized program can do that. Most of them are still thinking that even if they're in an Agile format, that they're going to pick some version of the release, someone that has some of the most critical features in it and they'll periodically select a version of their agile process to go and beta test.
00:16:56
Mario Sancho, Chief Delivery Officer at Centercode
So we’re… the conversations now is like you don't have to select which is the most critical one, but let us work in a process where you can test all your iterative releases, not just the ones that you think are going to impact or have the biggest impact across the board.
00:17:10
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yep, yep.
00:17:12
Chris Rader, VP of Marketing at Centercode
I would definitely love some more standardization in terms of the nomenclature. That would be super.
00:17:18
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
I know. Right?
00:17:19
Chris Rader, VP of Marketing at Centercode
From a marketing perspective, we have to go after terms like beta testing and user acceptance testing, and field testing, and all these different variations of the test. So a lot of those terms are still used, it really much depends on the type of company and whether or not they're more engineering focused or more marketing and product focused or UX focused.
00:17:38
Chris Rader, VP of Marketing at Centercode
And they like to change. Telecoms, like international companies call it different things. So just… Mario kind of pointed out there is something consistent across them and it's not the name, it's the real people, it's the real products and it's in their environment. So that kind of distinguishes whatever term they're using…saying like, okay, we can all kind of unite around that concept more than the name.
00:18:08
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Oh yeah. It is tough though. I think we should form a consortium and just solve this.
00:18:13
Chris Rader, VP of Marketing at Centercode
This is now the second time on the podcast. I've been told that we need to have a consortium out there.
00:18:17
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Have you?! Alright!
00:18:20
Chris Rader, VP of Marketing at Centercode
Yeah. It wasn't for that one specifically, someone was talking about… Yeah it was, It was one of the setup protocols. We were talking about TVs and I don't know if you've ever set up a TV and you have to log in to like your Netflix or anything like that and you gotta… this person says, Oh yeah, you can scan this one or you get to log into this thing or you don't…
00:18:41
Chris Rader, VP of Marketing at Centercode
You have to go figure it out yourself and type in a keyboard to figure out your. So it's like, Oh yeah, we got to get people together and do that. So yeah, we'll put it on the list of consortiums that we need to put together.
00:18:52
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
I'm down, sign me up.
00:18:54
Chris Rader, VP of Marketing at Centercode
The one thing I really enjoy... So I came from beta testing as well, like I did beta testing at a company. We did not have a tool. We used a lot of different tools. So you mentioned something, Sharon, is like once you get hooked on a good beta. Yeah, when I started my career, I went straight into beta testing.
00:19:15
Chris Rader, VP of Marketing at Centercode
I wasn't a beta tester. I was like a coordinator of beta. So I had an engineer above me. But I would just respond to all the feedback and get all that through. And then at one point in time I was tasked with trying to understand how the beta program was going overall. And I dug into it like, man, this engineer is doing it different on that product line.
00:19:36
Chris Rader, VP of Marketing at Centercode
And this engineer is doing it different on that product line, and I couldn't get that answer out of anybody. What is good and what is a good beta test or a good beta program as a whole? And I couldn't get a good measurement to tell me what was good because everyone was doing it slightly different. Now, have you, have you seen that?
00:19:54
Chris Rader, VP of Marketing at Centercode
And you've been at a lot of companies, So I feel like you have enough to share on this, but like what's your version of a good beta? And when you come into a company, what does the beta program typically look like if it's not a good beta program?
00:20:10
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So, typically it's nonexistent. That’s what I experience typically, and a good beta program is one where everybody finds value at the end of the day. I know that sounds kind of trite, but that's really what it boils down to. The engineers find value because it's a form of extended QA… the product people for that same reason like that.
00:20:41
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
But they're also getting, if you manage the beta program well, you get advocates. So then PR is happy because they get quotes at launch which normally get from real people, real customers, right? So they get hooked on the drug real fast. Similar in marketing, you know, you're doing essentially market validation as not I mean, it's not true market validation.
00:21:06
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
I don't want to pretend that, but it is… there is a significant element to that that is extremely helpful to the marketing department. So they look, hey, you know, it just goes down the line right where you can, if you do it well, you can predict a lot of stuff. The customer support organization you're putting together here are the likely call drivers, right?
00:21:31
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
They love you because they'll have to wait and get their knees scraped. They can already have articles written in advance and, you know, be ahead of the game. And same with even, you know, going back to product development. They already know, they can tell what it's going to be the dot release and they can start working on it.
00:21:47
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So even if you go to market with some known issues, you know, in advance, they're already working on the fixes. People complain, as everybody predicted. And then the next week you come out with the, you know, the fix for it and everybody's like, oh my God, they’re so responsive. And so the whole company looks brilliant. When really, it was just this secret weapon of doing a well-run beta.
00:22:11
Chris Rader, VP of Marketing at Centercode
And so I don't want to call you a peddler at this point, but you come into a company and there's no beta. How do you get them hooked? You like, you give them you give them something to try.
00:22:23
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, it is tough. So I… so here's the reality. If I get hired, that means there's at least one executive in the company who believes in the value of some form of user testing. And so I naturally have an ally and then the rest of the company probably doesn't want me to be there.
00:22:49
Chris Rader, VP of Marketing at Centercode
Why?
00:22:50
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
They feel threatened. It’s like, no, we write good code, why? And you’re gonna, you know, you get paran-... because it's your baby. You've been working on it for a year or whatever. And it's. It's scary. I get it. It's emotional. It's scary. Right? And so that that comes out in the way of, you know, diminishing what it could be. Do we really need this or we don't have time anymore for it?
00:23:18
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
What? You know but first things right?
00:23:20
Chris Rader, VP of Marketing at Centercode
Some resistance.
00:23:22
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah. This goes well. It's testing. Yeah, that's right. Yeah.
00:23:25
Mario Sancho, Chief Delivery Officer at Centercode
Guidelines. So we're good to go, marketing did its research, of course we developed a perfect product. Everybody’s gonna love it.
00:23:31
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, well, the problem that though is that so much time passes from the market research until you build it that you don't know if it's relevant still, right. And so it's like you know let's let's that kind of marketing plan but.
00:23:44
Chris Rader, VP of Marketing at Centercode
But someone has to ask the question right? So someone has the question, is it, it may not be that person who made it, but someone has that question.
00:23:52
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yes.
00:23:53
Mario Sancho, Chief Delivery Officer at Centercode
And usually how you go about it, Chris, is just status. Somebody has the question is identifying that person. Sharon talked about it and it's like, usually an executive, from my perspective that if I'm identifying the person that has doubt about everything that they've done, whether we want to say a simple question like what keeps you up at night? That's usually the way you get a beta program in to solve that first, that that first doubt that they have to make sure that we can put them at ease one way or the other through a properly run beta test.
00:24:25
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Oh, that's a great strategy. Yeah. Yeah. I normally like my, my strategies. It's similar. I like to go around and I have one on ones with people and as I just have one on ones with everybody when I first start at a place and really get to know people really get to understand where they're coming from, what their concerns are, what their beliefs are about user testing, whether it's UX testing or beta or whatever. And then I kind of explain the process and and that my role is to make them look brilliant. And we're going to do that behind the scenes within the walls of this company and even of the beta program. There's still like a you know, it feels like it's out there, but it's not really out there yet. It's not in the public eye yet.
00:25:16
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So there's still time to figure out how to respond to things that we may not have anticipated. So that's the first thing that I do when I go to a company is basically a sales job just to kind of warm people up and then and then there's still, you know, people kind of nod their heads and you can still see the doubt in their eyes.
00:25:34
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
It's like, okay, I guess we're doing this, you know. And then they get the results and they're like, Oh my gosh, yeah, they're the ones going to bat like, we definitely need to keep doing this. And it just always takes care of itself. And that's it's been my experience at every single company.
00:25:51
Chris Rader, VP of Marketing at Centercode
I have a good story. I won't tell it right now, but I have a good story for the opposite of someone who really… I want to ask you guys the question first, someone who didn't have doubt and it was proven that they needed to run a test. There was no doubt in anybody's mind at the company that they needed to do this and they threw every… they threw a lot of money at it to solve the problem. They saw beta, okay, that's going to solve this problem. Really my story has to do with products having to be returned. And the company had some pretty big problems and they needed to solve this before… right now. They needed to solve it right now. So have you had that experience of someone that did not have doubt they were certain they needed to do the testing? Like come fix this for us. Make this better.
00:26:36
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
No, but I can imagine like that. That to me, like I immediately smell… um, panic? And whenever there's panic at a company, (laughs) things get weird. And so I would like my response to that would be to defuse the panic and work on that in parallel would actually probably it sounds like they probably wouldn't need a beta program, but I think that's almost secondary to whatever led up to that moment.
00:27:10
Mario Sancho, Chief Delivery Officer at Centercode
For me that actually conjures something slightly different from what you're asking for is the person that wants to do a beta test or the company wants to do a beta test for what I call the wrong reasons. Well, the reason is we're going to sell this… our product to these beta testers.
00:27:25
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, oh that drives me crazy.
00:27:27
Mario Sancho, Chief Delivery Officer at Centercode
From my head originally. Oh, okay, yes. Our product is so awesome and we’re gonna do a beta test as a way to increase sales, and beta tests will do that. But not…you’re using and abusing your customers in that moment for what they're there for, to help you. It's not to sell them something, it's to get their honest feedback to help you make a better product.
00:27:51
Mario Sancho, Chief Delivery Officer at Centercode
So therefore you can go sell it to other people at a faster, higher rate than you would otherwise. So those are the ones…when you said that, that’s where that went, where you want to do a beta test…
00:28:02
Chris Rader, VP of Marketing at Centercode
What's funny is there's a lot of variations of what… what beta is solving for somebody, like we talked about the checkbox. I just need to get this out the door. I need to do this. There's the company that someone there has a question in their mind of - I don't know if I really trust this. Can you help me make sure that I can trust it and get some, you know, confidence boost?
00:28:24
Chris Rader, VP of Marketing at Centercode
The one I was talking about was the one with the “oh, crap” moment, we launched something and we need to pull it back because we released with something real bad in the field.
00:28:33
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Oh my gosh.
00:28:33
Chris Rader, VP of Marketing at Centercode
The one I'm talking about is a company shipped a product and the reviews were atrocious. It had a pretty big manufacturing blunder when they produced the units and they actually did pull back the units. They had bad reviews. It cost them hundreds of thousands of dollars in total to just bring back the units. They had to remanufacture the pieces of the broken unit.
00:28:56
Chris Rader, VP of Marketing at Centercode
And they came back and said “Please, our, we need a better program, please. We need everything in place to check before this product goes out the door because we can't spend that money again on another failed product release.”
00:29:09
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, that's an expensive lesson for sure. That's painful. That's painful.
00:29:16
Chris Rader, VP of Marketing at Centercode
Go ahead, Sharon.
00:29:18
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
When I started at Sling, we were just expanding internationally and we had, you know, this is a hardware device that connected to both your TV and your Internet. And it worked, you know, brilliantly in the U.S. going to Europe. And they're literally…we're about to launch, and thank goodness we did the beta because it turned out there is a network setting on home routers in Europe that has a value outside of the range for the U.S. And because we had coded for the U.S., nobody thought about it because nobody experienced it ever.
00:29:59
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
All the tests were running fine, right? So it literally was going to launch in a way where nobody could use the product, where it would be 100% fail rate. And it’s the simplest thing to correct too, right? Like one line of code and you're done, but luckily caught. So thank you beta!
00:30:18
Chris Rader, VP of Marketing at Centercode
That's the war stories that most people don't get to hear like when I say the story of someone that released it and it went out, it's like, oh yeah, there's money attached to it. You can explain it, but here's what we saved the company from doing.
00:30:31
Mario Sancho, Chief Delivery Officer at Centercode
Yes. What I used to call those when I was a beta manager…it was called the Golden Bugs. Something that was caught that you got to come and say “Stop production. Hey, we found something, we’re going to get it fixed. It’s going to be great.”
00:30:50
Mario Sancho, Chief Delivery Officer at Centercode
Those are the best. Those to me, those were the best. I was always hoping as a beta manager to find that… to find the thing that just one individual item that came out of it that saved everything, saved the company millions dollars, saved people jobs and careers in some cases, because it did what it needed to do, which is double check our work prior to going out.
00:31:14
Chris Rader, VP of Marketing at Centercode
Yeah, I mean, it's so hard to do it without that, right? Like how… double check, you probably didn't do a great enough check yourself just being a developer or a product manager or QA person. Like assuming that you're going to cover everything is a little wild. Yeah.
00:31:33
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
All right, Chris, you might be one of the only other people on the planet that's got the depth of experience in both beta and U.X. research.
00:31:46
Chris Rader, VP of Marketing at Centercode
There’s a few… There are a few out there.
00:31:48
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
No! Where are these people?
00:31:51
Chris Rader, VP of Marketing at Centercode
Because I know a couple. I know a couple I've worked with. And that's the only reason I know.
00:31:56
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Now, because I've been in more recent years, I've actually been focused more on UX research and when I interact with those folks, it's like beta is way over somewhere else. It's like zero interaction. And I was even at I was at this online, you know, meetup type event with I dunno, sizable, like 60 other people or something…all UX researchers and asked the question “oh so you know who's integrating beta as part of their… or overseeing it or knows about or in some way intersecting with beta. Chirp, chirp! Chirp, chirp!
00:32:38
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Finally, finally somebody goes “Umm, do you mean a longitudinal diary study?”
00:32:47
Chris Rader, VP of Marketing at Centercode
“An ethnographic study?” But yes, there's plenty of names for it.
00:32:52
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And I was shocked. And because to me, it's all under the same umbrella. So I'm curious to know, you know other people. This is great!
00:33:03
Chris Rader, VP of Marketing at Centercode
Yeah. It's interesting when you think about user research, it's users, right? So it's beta testing, reaching out to those people. It was a natural connection to me. You know, when I worked there, it was when I worked at Western Digital, I kind of started up in user research there. I picked up some books, I met with the UX manager and I was hooked on it.
00:33:24
Chris Rader, VP of Marketing at Centercode
The idea of psychology and thinking with users and all that stuff. And I was doing beta at the time, which was far more QA focused. So I think that's really the disconnect is it's, it's a separation and when the product is developed and when it hits each of these teams. User research and UX is design, the product isn't really fleshed out yet.
00:33:44
Chris Rader, VP of Marketing at Centercode
It's a lot of prototype stage. Some of that and it's in a slightly different area of teams until Agile really came in and it started like sort of blending everybody into one project group. And that's where I saw that connection. But user research should absolutely care about beta and the reason why I see it all the time is because UX talks to the people that are running beta.
00:34:06
Chris Rader, VP of Marketing at Centercode
So it's not always UX that are necessarily running them, but they have the survey experience, right? They know how to write a good survey. They know how to dig into a little bit of depth when they're talking to users and probe for some, you know, good responses and… yeah.
00:34:22
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah. Yep, yep.
00:34:24
Chris Rader, VP of Marketing at Centercode
To give some context, that's actually how I met Sharon from because I was doing user research. I was dabbling here at Centercode and I worked with Sharon on a couple of projects and we did some moderated in-lab studies. We brought some people in and we test out some products and just a natural connection.
00:34:41
Chris Rader, VP of Marketing at Centercode
She was already doing beta testing on those things and it was great flavor ads, if anything.
00:34:47
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah. So how does… what’s this methodology, this is cool. And then I think at…let's see that was with… we did that with Anki, I guess, we didn't I remember doing an out-of-box with you on that. That was great. That was very helpful.
00:35:03
Chris Rader, VP of Marketing at Centercode
And we hired one of those participants. I don't know if you know that.
00:35:06
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Did you?
00:35:07
Chris Rader, VP of Marketing at Centercode
Yes. Hailey! Our Betabound Community Manager participated in those studies. Yes. So she brought in her younger brother to give feedback. And she was just, she was there because we met through Betabound. That doesn't make me feel younger. But yeah, we hired one of those participants. It's interesting when you think about a user researcher, they have a connection to design, right, that's their resource. They produce the designs. And engineers have QA, and QA is the same connection.
00:35:44
Chris Rader, VP of Marketing at Centercode
And so those two are separated. So it's like that's why there's a somewhat of a rarity of what Sharon seeing is in that connection between both of them, because it has to be a good environment for a UX person to say, Hey, engineers, but I ran into these things that are not really my camp, they're your camp. And then that starts people pointing fingers.
00:36:03
Chris Rader, VP of Marketing at Centercode
And that's what I always felt when the tension of working with those teams early on: you're calling my baby ugly and you're not even supposed to be working on my baby like you work on yours and I'll work on mine. So it was always like, Don't bring your bugs to my area or don't bring your usability issues to mine.
00:36:23
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah. So, okay, so that brings up a good point. And that's something I also really want to talk to you about. I feel like the way organizations are set up where research is done in silos is less than ideal for probably most companies, and I'm sure there are exceptions, but I've experienced, you know, helping to build and being a part of a research group that funneled up into its own executive sponsor and included market research, user testing and beta all under the same roof.
00:37:07
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
We were one team. Now on a day to day basis, everybody's off, you know, working with their own teams face to face individually, but they're not reporting to that manager, which is kind of a you know, fox in the henhouse sort of situation, honestly. Right. So it's a little bit less biased. And also, there's so much cross collaboration at the end of the day when everybody is kind of back at their desk and like, oh I’m doing this and oh you found that you know result? Well I found this result and you can actually get a clearer picture of truth and the relevancy of that is I've seen more than once the marketing department deliver one report and the product department deliver another report, whether it's from user testing or beta or whatever, and the information conflicts.
00:37:56
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And nobody knows this. Nobody sees this except the executive staff, and they are left with having to decide who's telling the better truth. And they only have, you know, 10 minutes to decide it. And so they make a decision.
00:38:10
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And it's just that is the wrong way to go about it. Mm hmm. If you have the research, what I saw is that when research conflicted the actual researchers or who were on the front lines, who actually had talked to people who were or who had been, you know, buried in the numbers and did all the calculations, they can actually talk and figure out together what's truth and then have that reported up.
00:38:34
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
That's harder to do when those people report to different managers because then it gets territorial and personal versus when you're working on the same team, you’re working towards a common goal, which is fine tuning for the company. So the company can make the best decisions possible. So this is something that is I, you know, I think about a lot and I think companies will start to get smart and organized that way. I do hear of it a little bit, but not a lot. Do you see anything like that, Mario? In terms of, yeah, probably not right?
00:39:09
Mario Sancho, Chief Delivery Officer at Centercode
I haven’t, no. I can't say that I have, can’t think of an example.
00:39:18
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah it's because what happens typically particularly right is, when companies start or start you know they are siloed right? Marketing's over here doing their thing and you know, product’s are doing their thing and, and that makes sense. But I see a real opportunity for companies that are a bit more forward thinking to structure things a little bit differently to take advantage.
00:39:38
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And again, not it's not going to be relevant for all companies. I think especially if you're a really large corporation, that you just may be too entrenched...
00:39:46
Mario Sancho, Chief Delivery Officer at Centercode
Do you see that single manager then reporting directly to the CEO or like what? What?
00:39:54
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Yeah, ideally, yes, because.
00:39:55
Mario Sancho, Chief Delivery Officer at Centercode
The Chief Research Officer is really what it comes down to.
00:39:59
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So yeah, your research or insights or whatever, right? Because you're also I mean the other reality is you're getting a lot of information from customer support or client success, whatever. You know, if you're B2B or B2C, and also sometimes PR, because if PR is running a community, then they are naturally getting suggestions. So having somebody who's responsible for all those touchpoints between what you build and the people who use what you build and mining for information from all those touchpoints like that is smart and keeping that independent is even smarter.
00:40:45
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
So yes, that does require somebody who is funneled up at the highest level to be that truth keeper.
00:40:55
Chris Rader, VP of Marketing at Centercode
Yeah, definitely comes with maturity, right, of, of organizations and those programs because I mean you talked about it earlier sometimes the responsibility of research, user research, market research and beta testing actually falls to someone who's not it's not their title, not their role. So we see a QA engineer just have to pick up and do a beta test and like, Oh yeah, you know, I'm close to the product, I know what's broken or I know how to read this stuff.
00:41:22
Chris Rader, VP of Marketing at Centercode
And they run testing or a product manager needs to go out the door and they just kind of get tasked, which is kind of, you know, Luke's story of Centercode, being tasked with running a beta when he had no idea what it was or it's not his job responsibility. So if there's no one that's responsible for it and everyone kind of has to pick it up, it's hard to mature it because that person's probably done with it when they're done with it and that product's done and now it's up to the next product manager, QA manager, support person to run it the next time they have to, you know, get a product out the door or a new release or update.
00:41:55
Chris Rader, VP of Marketing at Centercode
So yeah, that's, that's kind of what I'm seeing is just it really truly depends on the maturity. Once they break through, say, product managers like man, that was a great beta I got, I got good results out of and they convinced their boss like we should, we should figure out some way to run a beta program for the company so someone get a resource because we can't keep doing this like ourselves or it's too cumbersome.
00:42:17
Chris Rader, VP of Marketing at Centercode
What I see is that a lot of times it's the tools that they have to use that makes it cumbersome, the lack of experience, training and all that fun stuff. But yeah, yeah. So it's definitely possible. I've seen those insights teams that are huge, that have market research, user research and every type of research you can imagine and aspects of beta testing and all that fun stuff, but definitely slightly more rare because I think immaturity is far more prevalent in these companies now.
00:42:48
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And interesting, cool.
00:42:50
Chris Rader, VP of Marketing at Centercode
Cool. So one thing I know we're going to get pretty close to the time here. One thing I wanted to ask both of you was what advice do you have to your younger selves coming into tech? This is the great standard question for everybody.
00:43:06
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
I hate this question. Yeah.
00:43:09
Chris Rader, VP of Marketing at Centercode
This question. Be hungry. Be thirsty.
00:43:12
Mario Sancho, Chief Delivery Officer at Centercode
I love this question. So I'll start. I'll kick it off. Give you some ideas, some time to think about it, Sharon. For me, tech is all about curiosity. Curiosity about the way the world works, the way things work. It’s not “I want to be in tech because I like playing video games or I like cameras... “ That's not what being in tech’s about..
00:43:31
Mario Sancho, Chief Delivery Officer at Centercode
It’s about being curious, curious about things or that doesn't mean you have to be so curious and be like the next Steve Jobs. Just curious enough. What are the settings on my camera? What are the…what can I do with this and why would I use it? Maybe I never will. But what do these buttons do and get a little dirty and find out like oops that was a bad idea…
00:44:04
Mario Sancho, Chief Delivery Officer at Centercode
Break things and then fix them again. You're curious, you’re going to be great at tech… to me it just comes down to that.
00:44:11
Chris Rader, VP of Marketing at Centercode
So your advice to yourself would be “be curious.”
00:44:16
Mario Sancho, Chief Delivery Officer at Centercode
Be curious, be curious.
00:44:17
Chris Rader, VP of Marketing at Centercode
I like it, I like it.
00:44:19
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
It's great. I can't top that. Mic drop. I think.
00:44:23
Chris Rader, VP of Marketing at Centercode
C’mon Sharon, you have advice for yourself.
00:44:23
Mario Sancho, Chief Delivery Officer at Centercode
C’mon please.
00:44:28
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Ugh.
00:44:28
Chris Rader, VP of Marketing at Centercode
Improv, or…?
00:44:29
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
More improv. Yeah. Well, okay, so the question for me specifically or like…
00:44:35
Chris Rader, VP of Marketing at Centercode
Yeah, if you were getting into tech and like, you're like, oh man, don't, don't do that or you should really do that more. So you just stepped into the tech world and you're there for some reason. You know your current form and you're able to talk to yourself. What do you...what advice are you going to give yourself?
00:44:53
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Well, I do think that most lay people who are not necessarily involved in the production of tech grossly underestimate what's required to make a tech product…grossly underestimate. So I think the corollary there is when you look around and you're like, why are we doing things this way? And like, why are we spending money on blah, blah, blah, give some credit, some leeway to the people who've been there for a while and might have more experience with it and ask questions, you know, for sure, because it's important to understand, you know, why things are happening.
00:45:46
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
And honestly, you know, no company is run perfectly. So sometimes it is the people coming in with truly fresh eyes who are able to see the inefficiencies and actually have some really good recommendations. But there's some give and take there, I think. I don't know. I think for myself, I mean, when I look back, like just career-wise, I think I've always been such a perfectionist and self-reliant and so, you know, this is outside of tech, right?
00:46:20
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
That's just general idea of where you have to be okay with asking for help. And okay with focusing on the most important things as opposed to everything. So I’ve gotten better about those things over the years. But it took me a while. But that's just any job anywhere in the industry. For tech specifically… yeah, that’s tough.
00:46:50
Mario Sancho, Chief Delivery Officer at Centercode
So, can I answer about yourself? Because that was my experience of all the years of working with you, it's always been your confidence in getting people to open up and speak up and getting people to collaborate together, frankly, like you said earlier, in a safe environment to make them look good. So that would be looking back and working with you over the years and it was like, oh, Sharon’s always amazing at getting the best out of everyone to create something amazing.
00:47:18
Chris Rader, VP of Marketing at Centercode
I had the same experience when I worked with Sharon and she gave me… I'm new at Centercode and I've done, you know, a handful of tests at Western Digital, like probably a dozen or so. She gave me a voice in her product’s line and she stood behind me while I was talking to people I didn't know at her company telling them that here are some of the problems that we saw and things that worked.
00:47:37
Chris Rader, VP of Marketing at Centercode
And she was there amplifying it. So she was, no we need to fix that, that's a problem. And by all means, I was just handing the results over, my opinions. And what I've seen observations of this, this new person that no one there has ever met. So I agree with that, Mario.
00:47:55
Sharon Rylander, Senior Director of UX Research and Strategy at Square Panda
Well, thank you, you did good work. It was well-deserved, both of you. So but thank you that go butter my temples so I can fit my head through the door.
00:48:05
Chris Rader, VP of Marketing at Centercode
Haven't done one of those in a while now.
