The Autistic Rant Hour

‍Gal: Welcome to The Autistic Rant Hour, the podcast where we rant and infodump about the aggravating, the inexplicable, the unjust and infuriating in all fields of life, one episode at a time. The Autistic Rant Hour is part of the Autistic Culture Podcast Network. I'm Dr. Gal: Schkolnik, pronouns: They/Them.
You know how they tell you that websites and apps collect your data and follow your every move, and then design their product to get the most out of you?
And you know how each and every time a digital product gets updated, it gets worse and harder to use?
How can both of these be true?
Well, let me get you in on a secret I've learned during seven years in product analytics: It's true that most, though not all, websites and apps collect your usage data. But most of them just let it rot in dashboards nobody looks at, and release major features and redesigns without performing any mathematically rigorous analysis of the user's response. My product analytics ex-colleague Carolina:, pronouns: She/They will join me on this episode to rant with all our accumulated frustrations about the sad, sad fact that most digital product providers really don't care about your data. And also just in general, about being autistic in tech.
You can also find Carolina: on Instagram @millennialspinster. I will add a link to the show notes.
Great to have you here, Carolina:.

Carolina: Thank you.

Gal: Language warning: This episode contains a couple of slightly adult words. Please see details in the show notes.
But before we begin, let me tell you about something we can actually do about one of those aggravating things. I'm sure you know that moment you walk into a new place and realize you have no idea how to get what you need. And the sensory environment is already overwhelming you. I'm founder of Aut2Aut, a registered nonprofit that provides free digital solutions and platforms by and for the autistic community. One of those is prepped.to, a website where you can find and upload sensory information and service instructions for all sorts of places, so folks can prepare and script before going there. prepped.to is totally free, without any membership fees or premium tiers. The web address is prepped.to 
prepped.to is only as useful as the locations that autistic folks like you and me upload. You can upload as little as one line of text or as much as a whole essay accompanied by images, a video tour and a sound sample, all depending on the info and spoons available to you. Another way you can help keep the lights on at prepped.to is by donating as little as one Euro a month, or by shopping at our print on demand shop where all the designs are #ActuallyAutistic. prepped.to -  know before you get there.

So let me just get started by explaining what product analytics even is. It's all about assessing and measuring the performance of a digital product, both as a whole and of each of its components in particular.
For example, if a company has a website or app: How many users visit or install them? How many users return after their first visit and how often? How do users respond to emails and notifications? Do they click on them? Do they do the thing they were supposed to do after clicking on them? Where do they come from when they visit the app or website? What do they click on? Do they do the stuff the company wants them to do? Do they get lost or drop off before they can do that? How do they react to changes to the app or website?
As an example, on prepped.to, I have installed a free plugin by Microsoft called Clarity (who sadly is not paying me to promote it) and this allows me to see, for example: Out of the users who got into the “Add Location” page, how many actually ended up adding a location? It doesn't tell me why though, and I'm not getting any personal information about them.

The thing is, to answer any of these questions, you need all sorts of technical stuff such as data storage and data visualization solutions, and we'll get to those later. But what you absolutely cannot do without is event tracking.
An event is anything a user does on your website or app, such as open a page, click on a link or button, fill in a form, or watch a video. To track those events, someone has to add tracking to the code of the website or app. This can be done by software engineers, aka programmers or developers or devs. Or you can use an out-of-the-box solution that allows non-engineers like me to inject the tracking events into the code. This is what I did at one of my jobs. When I got there, I was horrified to find out that there was no tracking in place. This actually happened to me on two occasions. So if you think that each and every app and website you visit are tracking your every action, well, not all of them do. Some of you might be relieved to hear that, because you don't want your actions to be tracked. But there's also a downside: If there is no event tracking, there can't be any product analytics, and things can't be improved according to how you, the user, behave on the website or app. Whoever builds it just throws things at you and hopes for the best. But the best is unlikely to be the outcome of simply not knowing what users do on the website.

In one of the places where there was no event tracking when I arrived, the moment I raised the importance of it, developers were allocated to creating, maintaining and storing tracking events. But in the other, the head developer was simply against it. He hated me from the first moment, not because of professional disagreements, but because being autistic, and not having heard about office politics up to that point, I just told him it was unacceptable to post humorous memes with naked women in them on the general channel in the office Slack. While his boss had to confirm I was correct in my assessment, probably to avoid a lawsuit, I was later approached by one of the other executives who told me that I shouldn't underestimate the effect of constantly pointing out sexist practices in the office. Being autistic, I didn't understand that he was asking me to stop doing that, and in fact, threatening to fire me if I didn't. Also being autistic, I didn't realize that I wasn't supposed to reply: “And you shouldn't underestimate the effect of constantly being infantilized and objectified your entire life.” Another thing I didn't realize at the time was that I was in fact, autistic.
Long story short, I ended up implementing the event tracking using a no code solution that allowed me to inject the tracking events directly into the website's code, and I documented it meticulously in Excel files. About a year in, after, according to rumor, the head developer announced that it was either him or me, I was fired. And once I was out of the way, he deleted all the tracking I painstakingly implemented. For years, this website had no tracking on it and also no product analyst. They just built it and never looked back. When eventually they realized that they actually did want to know what users were doing there, they hired a product analyst, who dug up my old documentation and implemented my tracking all over again.

Carolina: Oh yeah, absolutely no surprise there. And you know what's the worst part about that? That that whole time they could have recorded or stored the data of like what their users were doing. They didn't, because they just deleted your work and now they had to hire a whole new person. And then they only are going to have data and they're only going to have insights, you know, from that moment on. Had they left it as it was, by the moment that they hired a new person, they at least would have a base.

Gal: Yeah.

Carolina: And, yeah, people don't think about it because, yeah, because you only start getting information on it from the moment you actually do the work of implementing it. Because I think, yeah, like tracking is really like underestimated in like the product development process because it's not a step that, you know, engineers, developers, have like top of mind necessarily, because it doesn't really do anything to the things that they're trying to make happen in the sense that it's not part of that code that makes the function function.

Gal: Yeah. Like they're trying to make something work and look a certain way. Tracking it is not part of that.

Carolina: Exactly. It's not like a line within that code. No, it's like a separate one that is pretty short and straight to the point. But yeah, it's like an extra step that doesn't move anything inside of the product itself. And then its value is like an investment, you know? So you see it later.

Gal: Yes. It's also not necessarily relevant for their work, right? Like, because the analytics is something that later is going to be relevant for the product manager and the designers, but not necessarily to the developers. So like, they, they can't be asked to think about it on their own.

Carolina: Like definitely, yeah, I wouldn't expect that necessarily. But also from, you know, product managers and stuff, it's not top of mind and not in the way that it makes sense actually. Yeah. I was in a meeting just like yesterday or something and I informed like my manager that: Oh yeah, it turns out that they implemented the tracking for Android, but the tracking for iOS hasn't been implemented. Exactly. And then she was like, oh, I wish they would just like implement tracking correctly. Right? And I was like, yeah, that's kind of the point. But like, yeah, it's, it's like a buggy before the horse thing to no end and you're just like pushing the horse, and then the horse is like pushing the buggy reluctantly. It's kind of like that.

Gal: I can really visualize this horse now, exactly.

Carolina: And then you're trying not to get kicked.

Gal: Whoa!
And like, then there is the question of documentation because, like. I mean, okay, for it to be implemented correctly, someone has to decide what tracking will go where and what it will be structured like, so later it can be filterable and you can use it in queries and so on. But also once you decide that, you have to write it somewhere.

Carolina: It has to live somewhere.

Gal: Yeah, yeah, and someone has to find it there. What's your experience with that?

Carolina: Documentation inside of teams, I think, yeah, it's easier to accomplish than across teams. I've been in like analytics teams, where like we documented very well and I learned that from the team. And it was not only about having it there, but also like about learning about it. And I really like that attitude, but it's different in different teams. And also when you're sharing a document or a messaging system where like engineers and like where QA and with this and with the other and everybody leaves comments and then you go through like the comments.

Gal: Oh, no!

Carolina: The documentation, because maybe the comments are like an addition or a change to… so it's, it's like a rabbit hole. And then, you know, there's different uses for documentation. And in this case, it's like for the process, it should help ease the process and make it something you can reproduce. But it's hard because then nobody reads it. Nobody like wonders like, oh, how they did that last time. Let me look it up. Everyone wants to reinvent the wheel. I don't know.

Gal: Yeah. And also like different roles use different places to document. Like I would document it on an Excel file, but the developers wouldn't look at that, would they?

Carolina: No. Yeah, exactly.

Gal: They want it to be on the design in the Figma where it says what the feature should look like. To those who don't know it: Figma is basically a software where you can put designs and you can put comments on these designs or like little notes on them. And so what they want is the tracking to be in these little comments, in these little notes.

Carolina: And it's: imagine a pin board inside a computer, and then you have sticky notes where you write text, you have to type it. And then yeah, with your mouse, you put it there. And so that's Figma. And you can have like more advanced stuff for design itself, but for more process things, where more than one field is involved, it's just a bunch of post-its and like screen caps in a big square, looks like paint. I'm like: How are you gonna organize a process in something that kind of looks like Microsoft Paint? You know, like I need, I know it's because we're analysts and stuff and, but yeah, I need like rows and columns. I need coordinates.

‍

Gal: Exactly. I mean, because also like it has to be structured because it’s a structure

Carolina: Like a wiki, you know, like either columns and rows or something or like a wiki: that's necessary. And then that's another discussion, which feels like, you know, the, the buggy before the horse, which I think is like just, yeah, like at least for me, like a description of like being, you know, neurodiverse or like autistic at work sometimes because a lot of things feel like that because you're like, I'm following kind of like common sense and logic, and it's not necessarily what everybody's like following. Not that I'm right or anything, but, you know, like people want to work on things that have visibility. People want to work in things that represent like disruption or make big change. And I'm like: But does it make sense to do that? Let's just check. Let's just check.

Gal: Yeah, exactly. And I mean, also common sense: It's only common within the place that it's common in. So like if we have the same common sense because we're both autistic, then some other people who are neurotypical will have like a different common sense. I guess that's the whole double empathy problem. And I mean, for me, putting things in a table is the most visible and useful way to do it. But I guess some other people think that common sense is to put it in a sticky note next to the feature that it's tracking. So like, eventually you end up having both and having to like, whenever you change one, you have to change the other and then no one looks at it ever again.

Carolina: Figma is the bane of my existence, like work-wise right now. Well, that and Microsoft Teams, but like Figma on essence, like because of what it is. Like, I know, because I work with a table that's like black and white, and then I make like some charts and I accept like two types of charts that I make. Figma feels like Pinterest to me a little bit, you know?

Gal: Yeah.

Carolina: And it's all like about the aesthetic, which yeah, if you're a designer and you're working with like design stuff, of course that's your main priority, but not for me.

Gal: Like, yeah, you don't want to have to dig in like endless discussions in like some thread in a sticky note inside a design.

Carolina: Because you cannot create a table inside of that sticky note. I've tried.

Gal: Oh, no!
And I mean, that's not the end of the story. Once you have tracking, there is the question of how you're going to store it. Carolina, you have some opinions about that...

Carolina: Oh yeah, like in general, you know, if you're talking in general and you're giving like an intro to analytics or something, you say: Oh yeah, there's on premise, there is hybrid, there is cloud, but nowadays everything is cloud. And like, what I mean by those things like: on premise is like having things on your local computer. But yeah, but like macro level. A company that runs like an online business is going to need a lot of processors and also like a lot of like hard drives to keep all that information if they're tracking. So that would be on premise. And then pure cloud is like everything is there and the processing is there and it's all like virtual. It's virtual machines, virtual processors, virtual clusters, everything is done there, like pipelines. And like in my experience so far, like the switch has been like real fast to just doing everything like that, you know, on the cloud. And I think some things are not necessarily like needed like that, you know. They don't need all the ticks for which cloud is like better. It's like, oh, you know, available at the same time in different geographies. That's like an advantage of having stuff like on, on cloud and if different teams need to access it or like make it more secure. But things like, for example, the kind of like data that I work with that doesn't have any personal information, and it's only for like one team in one company, I don't know why we need to have stuff on cloud and pay a bunch of money for like front-end tracking.

Gal: Because what kind of thing do they charge you for when you have it on cloud?

Carolina: Like, I don't know, like the rates, but it's pretty expensive. And then I've had conversations with data engineers about creating a table with events from a function of an app that I work for. And so I need to create like a big table with events. And then there's like a line every time a person does anything on that part of the product. So there's a lot of rows. So it's a big table for every day for months in the back. Users only visit every few months or something. It's not a thing you log into every day. So they need to look back to see the performance of the thing. They said, like: Recreating the table going back, it was going to be like two thousand Euro.

Gal: Oh my God

Carolina: Like just that table.

Gal: That's wild. Would it be fair to say that it's kind of like the Airbnb of data where folks got lured in by simplicity and prices, only to find themselves locked in and the prices went up?

Carolina: Yeah, yeah, it kind of is. And then it drove everything up and now everything is expensive in comparison because like, they set up the prices and there's like four big companies that are in competition with each other. And yeah, they can just say like, oh, now processing is this much or your plan is this much. And I, I'm not saying like, oh, yeah, let's get rid of like, I don't know, cloud and whatnot. Like that's how the economy is set up basically. But I think, you know, like at a working level. At an operational level, there should be also like openness or more discussions of like, you know: Do we need everything to be part of the migration? Can we think of alternatives, especially if it's like analytics data that only like one team uses in one geography? I'm like, we should just have a hard drive at the office.

Gal: Yeah, you should be like able to maybe download that table once and then access it on premises.

Carolina: Exactly. Yeah, it should be like that. Yeah. It is… It is kind of like a scam.

Gal: Yeah. Basically you've given your entire data of your entire operation to someone.

Carolina: Exactly.

Gal: And now they're like holding you hostage.

‍

Carolina: Exactly. It could be over payment or like you're trying to, like, rework your contract. And also in terms of like the technology, like dependency, you know, like if you're building your whole thing on like Azure, now everyone gotta know Azure. And if you're going to do a migration, that's going to be like the biggest thing that's going to happen in like ten years.

Gal: Oh my God, yeah, I know: migration!

Carolina: And the thing is that you're only migrating, quote unquote, like you're not moving anything yourself. You're changing companies like, oh, we're moving from like AWS to the other one.

Gal: Yeah.

Carolina: Yeah. You need like so much orchestration, but it's just to make one private thing, system, work with the other and do the same thing they were doing before.

Gal: Yeah. Well, I guess they make it difficult for a reason. Exactly.

Carolina: And we're all just like, yeah, sure.

Gal: Okay, so now that we've ranted about infrastructure, we'll go on to talking about what we actually are meant to do with the data, but often can't. But this will be after the break.

We're back from the break. We told you about tracking user actions and storing the data. But all of this is for data-driven decision making.
There are all sorts of ways to make product decisions that are based on the data that tells us what users do on our product. One of the most important ones, in my opinion, anyway, is AB Testing, where you randomly divide your users into two groups, where one group is given a newer version of some part of the website or app, and the other still gets the old version. You then compare the performance of the two versions according to some metrics that you define in advance. For example, you may change the shape, color, or text of a button and check if more users click on it. Or you might add a pop up to encourage users to do an action and check if more users complete the action. As a precaution, you might also check whether more users left the website because the pop up annoyed them. Dividing your user base into two random groups and then serving each of them a different version of your product takes a certain programming effort that costs engineer hours. And these tests can run for weeks before you get statistically significant results, depending on how much traffic you have. But without running an AB Test, it's very hard to know how your users reacted to a change. Let's say you make a change to the product, and you want to avoid the cost of an AB Test. You might say: I'll just look at the way the metric changes after I release the new version. The thing is, many things can happen at the same time as releasing the new version. There might be something happening in the world that affects how users use your product, be it a tech scandal, Olympic Games, a pandemic or war. Your competitors might happen to release a new product or promotion at the same time. Another team in your own company might release other changes on other parts of the product, or the weather might change, causing users to spend more, or less, time on their phones. So you might see a change to your metric that has nothing to do with the change you made to the product. When you divide your user base into two random groups and release only one specific change to one of them, the sun is shining on both groups equally, and you can compare the way your metric performs in one group versus the other over the same time period.

Carolina: Yeah, I like what you said before that some product teams or companies just release things and like hope for the best or wish for the best.

Gal: Yeah.

Carolina: Because yeah, because I think, yeah, that kind of describes the, the scenario you mentioned here where they want to make changes and they have analytics at hand. There is tracking that can be implemented. That means you can do an AB Test and then you got like to convince, you know, stakeholders to do it or to do it for long enough, and then that it’s worth it. Yeah, I think that's really hard. And I think there's also like, kind of like the other side of it, of having, you know, a lot of like expectations, or point that they're expecting, you know, the data to get to at some point, instead of like try and do it. Like I know the place where I'm right now, I think we could be, you know, AB testing stuff. And they're like: Oh, no, we want to get to like the good point. We want to have like the new tracking and everything. It's like

Gal: Oh, like the data has to be perfect before you implement AB Testing.

Carolina: Yeah. And it's like: we're never gonna have everything like, you know, aligned, aligned, aligned because like, they're going to keep implementing changes in the product. Like we can’t mandate how the product is going to go just because how we want the data. That makes no sense. But I think something that is lacking in all of these scenarios is that people don't really want to sit down and just think about: What are we trying to prove? What are we trying to test? and How are we going to check for it? It's interesting that thinking: Oh yeah, we're going to find opportunities where we can improve, blah, blah, blah, and not looking at the current state of that part of the product or that feature, that metric is not even part of the plan.

Gal: Like they don't look at the product data from the feature before they decide to change the feature and how they're going to decide.

Carolina: Yeah, they don't think about it before they make it like a key result that they want.

Gal: Yeah.

Carolina: Like from the very minimum, I would say, requirement of knowing if enough people use it to merit changing it or improving it or whatever.

Gal: Yeah, I think like there's a lack of general understanding of what analytics even is and what the data can give you. And then what does it even mean to do an AB Test. I feel that in a lot of places where I worked, product analytics is kind of an afterthought to the product process. There's like something that they call “product trio”, composed of a product manager, a designer and an engineering manager, for example. 

Carolina: That's true. 

Gal: But where's the product analyst? They just sit amongst themselves without knowing what the users are doing, without knowing what the data is saying, and just deciding to do stuff. And then they just Slack you and tell you: Hey, can you AB test this? And it's like: What? What is it? What are we even talking about? Like: I've never heard of this feature until this moment!
I have endeavored in my career to like create workflows where like, I basically inserted myself into product teams as part of this process. And I told them I needed to be there from the very beginning because, first of all, just sitting in the meeting where they're deciding what to do (it's called discovery), and whenever they're like: Oh yeah, I wonder how many people click it? And I'm like in the background, like: Tick, tick, tick, tick.

Carolina: We could know, you know.

Gal: Finding out and then coming back and being like: Oh yeah, these many people use that. And then they're like: Oh, interesting. Then it changes their whole plan.
And then there is the thing about how to test something. So like, for example, if you change too many things at the same time, it's very hard to get any information from an AB Test. Let's imagine that you have a form and you change the order of the question, the way they are worded and the color of the text all at the same time, because you want it to look better, you want it to be more accessible, you want it to be more efficient or whatever, but you want all these things at the same time. And then you give half your population the new form and half the population the old form, and you get some kind of result out of the test. But you don't know if this result is because you changed the order, because you changed the wording, or because you changed the color, and now you'll never know it anymore. And so like, if you don't include a product analyst in this process of deciding how to change the form, it's going to be pretty limited what this product analyst can do later with the test. And like if the product analyst is there, then like they can tell you: Hey, actually we should do three different tests here. Or you can also do an A, B, C, D test where you have like one version with a new order, one version with a new order and the new wording and so on.

Carolina: Exactly. Yeah. You need to be able to attribute whatever change to a specific cause.

Gal: Yeah. You just test the reaction to whatever you're giving them. And then you can't really tell what it was. And this is something they don't understand.
And product managers are the ones who get you into this bind. But normally, at least the product managers I worked with, most of them, let's say, I don't know, eighty to ninety percent of them are open to this feedback and open to changing the workflows in a way that includes analytics from the start and that takes these things into account. Sometimes you'll get some pushback, sometimes they'll like argue with you. And of course, occasionally there'll be a douchey product manager that just doesn't want to hear anything that *he* didn't think about in advance.

Carolina: Or can like attribute to himself.

Gal: Exactly. But I mean, normally it's pretty okay, but it gets worse when you have to deal with like middle and senior management.

Carolina: Oh, yeah.

Gal: Because like very often you will get stuff like the CPO or CEO or whoever, and sometimes even the data lead themself, who kind of want you to do weird shit, like follow a metric and see if it changes. And then they like give it a fancy name to make it sound legit. Like they call it pre-post analysis. And it's like kind of: You're just telling me to look at how the metric is doing after I release the change, which is what we explained at length earlier why it's completely not a valid approach.
You know what, I respect it if someone says: I don't have the resources to look at the data. I'll just release stuff and hope for the best. But if they like tell me: Please analyze this and tell me from an analytical point of view, what is the result of releasing this change, and then they tell me: But do it with a pre-post analysis. I'm like: I can't, it's not a thing!

Carolina: You can't really.

Gal: And then also another thing that I really, really like when they say, joke joke sarcasm: Okay, this test has been running for really long.
So like something I should probably clarify is that in order to get statistically significant results of a test, you have to have a certain sample size. So what does it even mean? Let's imagine that I have two groups of ten people, and I give each of them a version of a button. And I want to see how many people out of the group click the button. If, I don't know, someone in the other group happened to have a headache that day, then maybe in one group five people will click the button and in the other six people, and that will be like a twenty percent, or ten percent points, difference between these groups. But it's just because someone had a headache. Okay. So like if you have a very small group of people you test whatever you changed on, you're very likely to get some results that aren't actually true, but they're just kind of noise. And in order to know to a certain degree of certainty, which the common one would be: one out of twenty such tests will be an error, or your accuracy is ninety five percent, then you just have to have a pretty big group of users seeing this stuff.

Carolina: Exactly, like you need to have enough people to test your button where at least one, or minimum, having like a headache or a broken finger that wouldn't let him press the button, wouldn't, you know, move the needle enough that you're gonna make the wrong decision. That's yeah, you need to have enough, you know, for a sample. So now we know that ten is not enough. And yeah. And it's, it's, it's hard to explain.

Gal: It's hard to explain. And then it also takes a really long time because sometimes you don't have a lot of traffic. I worked at a project where we had millions of users a day, and that was amazing because I could just test for one week on one percent of the users and get amazingly statistically significant results. And I worked with products where you had hundreds of users every week, and then you might have to wait a month or even two. And then some big shot from the C-suite comes to you and tells you, so give me statistically insignificant results. And I'm like, you can just go and toss a coin yourself and tell yourself: This is the result. Like, you hired me to give you mathematically relevant results, like, analytics. That actually means something. You're paying me a lot of money to do that. And now you're asking me to just guess like, I don’t know what you want from me! I don't know, I guess some people have this capability of like some CEO or CPO is coming to them and is like: Give me the result now! And then they just say something, I can't do it. I'm like, I have, you know, my professional integrity. I can't tell a lie. I don't know, but, I mean, yeah, maybe it's a bit of a top down versus bottom up processing situation. Like we come from this autistic mindset where we're like, okay, let's look at all the information and then build our understanding of the situation. And they come from like, these are all the beliefs I have, and let's just implement them without looking at the data.

Carolina: And I'm like: No, that's so much work! You know what, I love parameters. That's my favorite thing for everything. You know, like socially, work-ly, anything just to know, you know, like: What's the max and minimum the temperature is going to be that day, so I know how to dress. Yeah. Like, yeah, it's just easier. Yeah. I don't know if it's good or something because it can also become like the behavior of like taking too much on yourself, you know, mhm, like responsibility or work and then maybe work that is not even necessary or something because of, you know, you want to do it right for your principles. So yeah, I'm trying to find that balance and also not kind of like catastrophize because like, you know, I'm a very like literal thinker. So they were saying: We're going to do this project. And I was like: Oh my God, it's going to be a whole thing. It's going to be so much work. And now we have to do discovery and this and that. And we had the workshop, but we haven't met since. Yeah, there's no follow up. And I keep forgetting that, you know, and then I'm worrying because…

Gal: Like, you know, what I would do? I would dig my own grave by being the follow up.

Carolina: Yeah, I'm, I'm trying to shut up because…

Gal: It's so hard to shut up.

Carolina: Yeah. Like trying to hold back a little bit on the proactive part of me during these times kind of helps you just, you have your contribution to make. So yeah, I'm going to help from that direction. I'm trying to do like that part now. Yeah.

Gal: And what about all those AI features for the users? Because like, I remember when AI was really just beginning and I was still in this field. They kept bringing out all sorts of like AI features that no one wanted. And everyone was like, oh, this is so cool. This is AI. Everyone wants AI. Now we have to have AI in our product, even if it's not a product that lends itself easily to AI, or even if you don't have the capabilities in the team to build the AI that the users actually want. And then they just also say, this is a strategic bet. We're not going to AB test it because we need to have AI. So it doesn't matter if the users are using it or not. And then if you do end up doing a test, no one is using it. They put so much effort into creating it and no one is using it. And then because they put so much effort into it, they just keep it there, even though the users aren't using it. It's like the sunk cost fallacy galore.

Carolina: Exactly. Yeah. Like buggy before horse and sunk cost fallacy is like the two banes of my existence in the tech world. There's always this pressure to be like, oh yeah, we got to do the next thing and blah, blah, blah. And yeah, sometimes they hold on too hard and then, you know, implement tools that nobody really wants and spend money on that. And then they're like, oh, no, we can't give you like raises we just bought copilot for everybody. Visibility is very important in the business. So everything you're doing, you're gonna hype and you can't drop it when you hyped it so much.

‍

Gal: Yeah, there's that too.

Carolina: Exactly. So then they're not going to say, oh yeah, we messed it up. Or probably they're going to wait too long, you know? So you're spending time paying for a tool for the employees that we don't really use or doesn't really meet our needs, but it's like the new thing, it has AI or something. So now everybody has to use it. Or like something common for analysts, it's like the visualization tool, like we don't get to pick it.

Gal: Oh my God, I have to rant a bit

Carolina: Yeah

Gal: right here about the visualization tool. It's very common. Everyone uses it.

Carolina: Drag it!

Gal: I feel I should probably not say its name 

Carolina: Exactly, just say “that one”.

Gal: because I don't want to get sued. It's a very common data visualization tool that is supposed to be drag and drop. And so like the idea is that you have a table and maybe another table and you, or like a column in a table, and just drag and drop it. And then you magically get, you know, like some chart or some other table. And you do get a chart or some other table! But then if you like, try to calculate the same thing that you were trying to do by hand just to make sure, because you should do that, you didn't actually get what you thought you were going to get. And then it turns out you have to learn a whole programming language just to write the equations, so to speak, to give you the result that you want. But it's also a proprietary coding language, and so it's not useful for you. And even when you do learn how to do this programming, invariably it's not going to give you what you thought it would give you. And there's a whole entire community of people who use these products that post questions to each other: I did that, why did I get that? How do I get this? Why did I do..? And like, have to like, really like send each other files with their calculations to, like, help each other. And then there are like the kind of the main honchos of this, like it's.

Carolina: Like Reddit, but just for like that visualization.

Gal: Exactly. And then they're like: Yeah, of course, you forgot to do this and that. And it's like, why did I have to do that?

Carolina: Exactly. That didn't make sense to begin with.

Gal: It's so terrible. And like, I actually, at some point I realized I spent so much time on this. You know, back and forth and trying to figure it out and getting the wrong numbers and all that. I realized that it would take me less time to learn Python from scratch. 

Carolina: It’s more useful, yeah

Gal: Exactly. And it's like a more general thing. You can use it everywhere. You can use it for all sorts of things. And it took me less time to learn this whole programming language, Python from scratch, just to create a dashboard, one dashboard, than creating this one dashboard on that fancy data visualization software that everyone has.

Carolina: Exactly, the quote unquote, drag and drop.

Gal: Drag and drop dead.

Carolina: Exactly. That's so good. The thing is that it's not just that it's a black box and it doesn't really let you see how it calculated the thing. You know, like at your home growing up, you had this old, I dunno, radio or like washing machine or something, that is like, it's about to die, but we can't replace it. So to wash clothes, yeah, we had that in my house, we had to fill it with a hose. And then, you know, you got to do all these special things like, oh, no, like this doesn't work. So we gotta, you know, put like broomstick here and blah, blah, blah. All things that are technically unnecessary in the action of washing clothes, but because you have this garbage washing machine, you have to do all these extra things to get sort of the same result.

Gal: It's such a great analogy to that thing.

Carolina: Yeah.

Gal: And then there's the question of the products themselves. When you're working in tech, sometimes you're working on a product that nobody needs. You work on a product that like, for example, has its biggest sale push in the beginning of the year, where people make their New Year's resolutions, for example, like a gym or something, where you sell everyone subscriptions to the gym in January because they're sure they're going to go to the gym now. And then they never go to the gym. So like, there are products like that.

Carolina: And you're counting on that. That's part of your strategy.

Gal: Exactly. And then like, you kind of spend the whole year trying to make this product more attractive or more useful or whatever for people who actually buy it to not use it. At least this kind of thing maybe has some use to the couple of people who actually stick with it. And then like you have other stuff, which is just like selling you stuff you don't need and just destroy the planet or tempting you into betting and gaming expenses. And like, you can work on such bullshit products that then turn your job into a bullshit job because while your job might be important for the product. The product itself isn't important. So like that makes it even worse somehow.

Carolina: Yeah it does. That's why, I don't know, I don't enjoy if the product of the company you're part of, like the main thing is like, you know, acquisition of like new users and stuff because yeah, that's going to be very driven in that direction. Like the example of the gym. This just convinced them enough to get them in. And if they have a good time, bad time, nobody cares. And then that type of business also, I feel, places like kind of less importance on like the product experience, you know, because you just want to trap them like subscription based things that are not like a necessity. Yeah, it feels kind of like pointless sometimes. Your work, something that helps me is just working on maybe a product that is like useful or needed, you know, like the app for like your electric bill. That sounds good. Like that would be good for me because it's kind of neutral, you know, people have to use it anyway. Yeah, okay. They have to choose what company. They choose yours. You give a good service, but it's not like you got to get them. You know, like a game or, you know, other types of app. That makes a big difference to me.

Gal: Yes. Like I kind of have this dream of working in something that makes total sense. Like, I don't know, selling food or something, like something people actually need. Yeah. I think the electric bill thing makes sense. Banking. I don't know, I'm not thrilled about working for a bank, but at least people actually do need that.

Carolina: Yeah, in the sense at least of like, yeah, the digital product that I'm working on. It's something that people are probably gonna use either way, be it the one I work for or another one. Not, you know, I necessarily like the values of banks themselves and how they behave, but at least in terms of, yeah, we're trying to get people to use this app because they kind of have to use it anyway. And so we're making it the best possible. Instead of like, you know, trying to convince people that they really want to use this and get this, you know, pay for this, it's harder for me to like work with that kind of thing. Like for a game app or something, because I wouldn't take it seriously, because it's not like, yeah, it's serious because I gotta pay rent. But then beyond that, it's not even serious because it's like: y'all are just pushing ads and stuff. So like you're just trying to get kids to gamble. Like, why do I care? This is not a hospital. Like, I feel like I gotta have some grounding. That's also why I like product.

Gal: Yeah. Well, I mean: a) I wouldn't work for a gaming app, and b) I would take it seriously no matter what it is. I think that's one of my autistic challenges.

Carolina: Yeah. Like I would too. It's just. Yeah. But like the dissonance would mess me up. That's the thing. Yeah, it has before, like, having to take it like so serious, like: oh, no, this release and blah blah blah. Or, for example, when they have those expectations that you have all the answers, all the insights automatically or without a proper process being like: Oh, we need to know who are our power players or something, you know, and do like a regression to see how long they're going to stay in the game. I don't know, something that only half makes sense, but sounds like something they would request. So I'm stressing about how am I going to get this done because I don't have like the resources, but it's also for like a game that is not like educational or nothing. It's probably one of the eight million candy crushes or like that lady with, you know, like the broken house and the snow coming in. One of the copies, not even the original. And so like that dissonance of like: We don't work at a hospital, Julian! Don't be like: Oh, if we don't find out who's the power player, everything's going to burn down. Like, don't act like that. So yeah, like that thing messes with me. and I think it's part of Neurodivergence.

Gal: Yeah, well, I guess, I mean, it is important for the people who make money out of the power players, right? Their house will burn because they bought a house they couldn't afford, and now they have to make a lot of money out of in-app purchases or whatever. And you're there to like, line their pockets.

Carolina: Exactly. But I'm saying like in the context where, yeah, they want to know who the power players are and they come with like: We just define power players as like the longest sessions, for example, and then they come to you with like a nonsense request that it's really gonna make it harder for you to do the job. And then it's also for like a weird game. That would be too much.

Gal: Yeah. And I mean, there are the nonsense requests and then there are the requests that just change every year. Like talking about bane of my existence. This was my bane of my existence. Like there is something called OKRs.

Carolina: Oh my God. 

Gal: Which, right? Which is like: The O stands for objective, which is like generally what we're trying to achieve. For example, we want users to develop a habit using our app, for good or bad reasons. And the KRs means like: How are we going to measure whether they're going in the right direction? For example: Are they coming back every week? Are they spending ten minutes a day, you know, this kind of thing? And the thing is that, okay, very well, there is some strategy. There is a CEO, there's like the whole C-suite or… deciding where this product is going, what is our objective and how we want to measure them. Great. That happens in the beginning of the year. And then, you know, they tell you what these metrics are that you have to focus on, and then you realize the tracking is missing. You implement the tracking. You start thinking how you're going to measure it. You start creating the calculations, the queries, all the stuff you need. You create the tables. You start building the dashboards. The product teams are starting to build stuff that is going to bring you towards this objective and towards increasing these metrics, and they start implementing and testing, and you're starting to do AB Tests with these things. And then the year ends and you have your initial results from trying to direct your product in that direction that this objective was set for. And then they just go and change the objective! They just, the stakeholders have a meeting, or the C-suite is having a meeting and they kind of strategize and go on a retreat or God knows what

Carolina: Had an offsite.

Gal: An offsite retreat.

Carolina: Took a bunch of pictures, paid some like, American like with a beard that does improv, to like do a workshop of like improv for strategic wins or something.

Gal: Exactly.

Carolina: Pay him your whole salary.

Gal: Drank too much, didn't sleep at night. The next day got up and then they were like: Okay, this is our new objective and these are the metrics. And then they come back to you. And then you just basically have to take all the work that you did through the year working towards that objective that was so important a year ago, and just throw it in the trash, because now there's a new objective and there are new metrics to measure it. And now you have to start all over again. And this really messes with my expectation sensitivity, with my need to know what's going to happen, with my need to plan, and also with my kind of need to go deep into stuff because like, actually sometimes some of these objectives are actually good and interesting.

Carolina: When you like one and they change it. It's like, oh, come on.

Gal: True!

Carolina: It's like when you're like, oh, this kind of makes sense. And they're like, no, we're not doing anymore, it’s like: Oh, shit!

Gal: Exactly. And like, how dare they? This was a good objective. And now we're just like, and we got interesting results. And I got excited about it and I wanted to get deeper into it. And then they're like, no, no, let's get AI.

Carolina: Exactly. AI: one OKR, the whole OKR is AI.

Both: Yeah. Yeah.

Gal: Like and like the feeling for me, I have also a metaphor. It's not with a horse, but it's also with a means of transportation and some animals. So bear with me. So my metaphor is: It's like a big ship, like a cruiseliner or one of those cargo ships. It's like a huge ship. It has a lot of staff and the staff is great. Like everyone you're working with in the trenches or in the, you know, lower decks, are great, like the Mechanics, the engineers, the cooks. In the case of the tech company, the product managers, the developers, the other analysts, the designers, everyone is awesome. But unfortunately, this ship is being navigated by a bunch of drunk monkeys who have taken over the bridge. So they just kind of change course randomly and you're like, just trying to hold on to dear life down there, and just deal with these frequent changes and still run the ship so you don't like drown. And that's what working in tech feels like.

I'm going to take a minute of your time to remind you about prepped.to, a website where you can find and upload sensory information and service instructions for all sorts of places, so folks can prepare and script before going there. prepped.to is totally free without any membership fees or premium tiers. The web address is prepped.to . prepped.to is only as useful as the locations that autistic folks like you and me upload. You can upload as little as one line of text or as much as a whole essay accompanied by images, a video tour and a sound sample, all depending on the info and spoons available to you. Another way you can help keep the lights on at prepped.to is by donating as little as one euro a month, or by shopping at our print on demand shop where all the designs are #ActuallyAutistic. prepped.to - Know before you get there!

So one last question: Is there something actually worse than product analytics?

Carolina: Oh, let me tell you something that is way worse. I'll never do that. It’s being like a marketing analyst.

Gal: A marketing analyst.

Carolina: Marketing analyst, because it's all the things that we have been complaining about. But just like, you know, the way we complained about them was like putting them in a shelf. And then being like a marketing analyst is having all those same problems, but you just like took an axe, chop the shelf and everything is in pieces and on the floor. And then they tell you you gotta pick random pieces like a piece of wood and a piece of book. And then that's a new book and they're like: This is a book, put it back in the shelf. And I'm like, how?

Gal: But why, why, why is it like that? I actually didn't work in that.

Carolina: Because there's a lot of stuff about attribution. 

Gal: What's attribution? 

Carolina: Like attributing the cause of an action that's outside of your visibility to one of like the pre-established like categories that they have in like the marketing analytics: If it's online, if it's an email, if it came organic, quote unquote, that would mean if someone Googles electric bill app and they click on it and then they're like, oh, I'm going to sign up because like, it looks like it's what I want. That would be like an organic subscriber or lead.

And everything in marketing analytics is like in terms of leads and like a lead is anything that can become a conversion.

Gal: A sale?

Carolina: A sale. Yeah. If you're selling things, yeah, a sale. If you want them to do five minutes with your app every day. You know, the first time they do that or something like that. But the way that they say what costs a lead and how many people or how many actions equate one lead follows no logic. So they can say that: Uh, we had a campaign for Facebook and this user probably came from Facebook, even though this user came from Google. But at the same time, we had a campaign on Facebook and they probably came from Facebook. So we're gonna attribute the same user. We can only make one sale out of that user to both those categories. And that makes no sense.

Gal: Yeah. So like basically attribution is to figure out where the user came from to your product.

Carolina: Yeah. Out of all the different, you know, channels or means that you're doing marketing.

Gal: Yeah. And then it's very hard because one user can be exposed to more than one channel. And also you don't really control the tracking, right?

Carolina: Not for, for marketing, not anything that happens outside. You're dependent on like the logic and the software of like a third party.

Gal: Of the platform where it was.

Carolina: Exactly. And it's probably, you know, like, like Google ads, Google Analytics, that's the main thing. And it's, it's also like a black box. Yeah. Like online marketing, the way I understand it on the side of like the marketers, it's like, yeah, you're going to put a bunch of money, like a bunch of money. Also, that is so much money, so much visibility that there's a lot of pressure in the like weird numbers that you don't understand that you have to report on because it's like fifty thousand dollars or something that you see in like the counter for like, some like Facebook campaign that I don't even know what it is, you know? But it's, it's big numbers. It's very expensive. And so they're just seeing how many clicks something generated according to Google ads, how many conversions and blah, blah. But all of those like what a click is, what a conversion is, is already predefined.

Gal: Okay. So like, unlike the situation where you design the tracking and you have trouble documenting it and people following it and implementing it, in this case, like you don't design it and you don't control it and you don't know what it means exactly. And then like, so basically, like you need to be a good bullshitter for working in this.

Carolina: Definitely. Yeah. Because you need to tell them like, oh, yeah, this thing you put 50k in was the best thing ever.

Gal: So yeah, not for autistics.

Carolina: Oh no, no, no, that, that broke me. Like I was like: What do you mean you're gonna count one person twice? Because they were like: Oh, yeah, we're gonna attribute this lead also to Facebook because we put a lot of money into a Facebook campaign. So yeah, they probably saw it. And I was like, but that's one person! And just to put into context, the job that I was doing, this ad was like a real-estate thing. So it is kind of like one person, one sale, you know, because it's full on houses. Yeah. Like if it's like a t-shirt shop or something, oh yeah, we can, maybe they bought ten t-shirts, you know, we can spread the attributions. But no, one lead can buy one house. And also sometimes you don't, you don't have like the visibility of whether they actually were exposed to like that campaign or something. It's just an assumption. A lot of assumptions.

Gal: Okay.

Carolina: Yeah.

Gal: I mean, if you want to hear more about assumptions in places you don't expect, such as in exact sciences, there is also an episode about that. But I don't know if I'm glad to hear there's something worse than product analytics, but at least it's nice to know that it's good in comparison to some things.

Carolina: Exactly. Yeah. To end on a positive note. Also, what's good, at least for me about working like product analytics, it's coming back to like that, you know, there's a way of having like a foot on the ground in the sense of you can only count, you know, one person clicking the button once. And then I like that, that makes it like more solid in terms of making, making sense for me. And then it's about people, you know, people doing things. At the end of the day, we're counting people. That's kind of fun.

Gal: Yeah, it's fun seeing what makes them tick and also making the product better for them. And I think on this note, we're going to wrap up. Thank you so much for being here, Carolina.

Carolina: Thank you. Super fun.

Gal: Super fun. It was fun working with you. And now it has been fun recording with you. 

And thank you to our listeners for listening to all this tech babble. I'm Dr. Gal Schkolnik, pronouns: They/Them, and I was joined today by Carolina, pronouns: She/They. You can also find Carolina on Instagram @millennialspinster. Please find the link in the show notes. 

This episode was produced and edited by yours truly. Theme music by Lir Lutau Shahar. Lir is a composer and sound designer who loves to make whimsical and magical music. Listen to more of seas stuff at @lir_lurim on YouTube or SoundCloud. For collaborations, see contact in the show notes.

‍

Rey: I actually don't mind podcast trailers, but I hate the fake happy music that is often used in them.

‍

Gal: Being autistic in a holistic world can have its moments of awe and wonder, but it can also have a lot of moments of just UGH. I'm At2Aut founder Dr. Gal Schkolnik, pronouns: they/them, host of the new podcast The Autistic Rant Hour on the Autistic Culture Podcast Network. And my little rant right now is about podcast trailers! They tend to start off with some discordant music that has nothing to do with the podcast itself, followed by either a breathlessly excited announcement or worse, snippets from the podcast, disembodied sentences that just jump at me with no warning and no explanation. In my podcast, The Autistic Rant Hour, I spent an hour ranting and infodumping on my own and with guests about the aggravating, the inexplicable, the unjust, and the infuriating, in all fields of life, one episode at a time. So for this trailer, I wanted to ask my guests how they felt about podcast trailers. You've already heard the opinion of Ray Rissanen about trailer music. How about you, Dr. Mary Sims? How do you feel about podcast trailers?

‍

Mary: Well, Gal, you know that I'm a stickler for accuracy. So when a podcast trailer comes on, my first reaction is fight or flight to protect my sensory avoidant self. This means my stomach tightens and I get very alert.

‍

Gal: Oh yeah, I totally know the feeling. I rarely even manage to survive the couple of minutes they take. So dear listener, you can totally skip this trailer altogether and just tune in to the real thing.

‍

Mary: The next thing that happens is that my autonomy drive kicks in and I start scanning the message for any hint of a demand being made on my time, energy or attention. I usually just can't wait until the podcast trailer is over. But if the message is clear, useful, and sincere, then I get this nice little dopamine rush.

‍

Gal: Okay, so they can also be done well, it seems. Carolina, what do you think?

‍

Carolina: What I like in a podcast trailer is like to hear who's going to be part of it and then hear why they're like excited about it. I don't need to hear, you know: Oh, it's so and so from this other podcast and creator of this and that, you know, I want to know what you're doing now and why you like it.

‍

Gal: Okay. So let me tell you what you can expect from this podcast's first season. I will talk about product analytics and being autistic in tech with my ex-colleague, Carolina, tell you about the bureaucratic nightmare of founding my nonprofit Aut2Aut, discuss medical gaslighting with disability activist Simo_tier, and mental health services with artist Ray Rissanen. With neurologist Dr. Mary Sims, we'll ask ourselves: Exact sciences, are they? And the last episode will be about accessibility across disabilities with allistic but disabled activist and filmmaker Tal(y) Wozner. Tal(y), what do you think about podcast trailers?

‍

Tal(y): I actually like trailers. It's cool that I get a chance to hear somebody's voice and see if it's annoying. And also, I don't know, get kind of a feel of what I'm looking at.

‍

Gal: Okay, so if you didn't find our voices annoying and maybe even enjoyed this little rant about podcast trailers, tune in to the Autistic Rant Hour on the Autistic Culture Podcast Network starting June 22, or wait for July 23 to binge it all in one go. 

‍