Lessons

Lesson Info

Homework And Coding Assessments

So next up, I'm gonna talk about homework and coding projects, So I had a also start with it. So when I have major concerns about Homer projects and this is where I start, people start to disagree with me a lot. But I have a lot of concern that Homer projects and the reasons that you talk to a lot of candidates, and I am very much empathetic to their side, which is that it creates this opportunity for exploitation. So I had a friend who was applying for a big company, and he got a project, no homework project. And I don't know if the company had idea of what this would. How long does that take? This is just a good example of why I think it goes the wrong, but, you know, they gave him this project and it was basically popular. Curious. You have a list of times of different appointments, and you need to lay that out on a calendar so you might have three appointments that overlap in time. So you to like, lay it out to be 1/3 a 3rd 1/3 of the column so ends up actually being a little bit a...

ll cosmically turkey, turkey, and he end up spending 12 hours to do this project. Now he's a single guy. No kids find whatever but people who with with, like, family responsibilities or whatever, That's a lot of time they're spending. But then, no, I didn't take 12 hours on time. He went and talked to all his friends about. Is this the best algorithm they did co drew. Use the bed in the dat time. Not only had he spent an absurd amount of time, but what he actually submitted was probably not really truly representing his own work. And there's certainly other candidates who are gonna be a lot less ethical who will just give it out to their friend to do so in the end Today, after all this time he spent in the company of probably spent 15 minutes giving a thumbs up thumbs down and inviting him on campus. That's not really fair. It's just in. The problem with homework is that it requires a little bit of time investment company, but very little for how much candidates and was spending on it, and that ends up being kind of problematic. It's very easy for those time to get wildly out of whack, to acquire, kind of to do 10 20 hours of work on that becomes really unfair to canceled, different your responsibilities. And frankly, if you're he went through this because this was the company he really, really wanted to join. But if your company who maybe doesn't have that prestige, you're gonna lose like Candace that way. So I have major concerns that Homer projects for that. So the week, that's what I want to distinguish this a little bit from coding assessments. So Homer Project is go build this. It's, you know, do this big big picture assignment. Code assessment is more like a test like that, you see on Hacker Ring. So it's usually, you know, no more than two hours. It's much. It's much more time constrained, and it's also, you know, generally more automated. So by shrinking the out of time, we fix a lot of those exploitation issues we can just get kind of take advantage of, and even in vertically they still get taken matchup. So I want to make that distinction that the symmetries of those two things. So there's benefits, though, for mark that said, You know, they're big, it's it is very practical. You get to see someone's real life very much real world skills, whereas the code assassin that's much shorter. You're not getting as much of a code sample getting very riel, real world skills. There's also, you know, some candids do really, really like it. The cans who hate the album interviews often like these kind of things. Um, so that's that's nice. And there's there can be less cheating eso its new some cans will just give it to their friend to Dio, but most candidates will do it largely themselves, whereas a code assessment test that's much shorter. It's much easier just to farm out that work to friend. Uh. The exception, though, is when you have a project that really involves algorithmic thing, eso. When that happens, then people may rely a lot on other Resource Is, and you're not really going to see if their their thought process. But if you just want to see kind of code and is their code good, it it'll kind of get you some data there, but you still can't rely on it. Too much problem, though, is that you said that it's very real world, but it's very world right now. It's not real world with Will they be good in 36 months? Necessarily, the summer could have really sloppy style and you think, rather than they're not a good soft engineer. But had they just haven't worked an environment that you cared about style, they never had that before. They didn't know, and you might think that they're much worse than they really are. It's not say that their style doesn't matter at all, but you may be unfairly or unreasonably biased against them. So it's very much real world. Not have how good they'll be in 36 months and then also you. It does take some engineering time to evaluate, and it doesn't take an enormous time. So, as I said, you know might take 15 2030 minutes to value this project against the can. It's 12 hours on it, but it's still taking time, and that means you have to go find an engineer to go review that thing, and then you're creating Ball Mac and they don't get back on their busy with other stuff causes. Interviewer engineers are often don't like to interview anyway. Now, when this just like oh, please review this as soon as possible. It's not a do this right now. You cry, start creams black, and you start making delays, having delays in process. This portion work loaded, as I said, and it just scales poorly for candidate. If they are interviewing with five of the companies, not everybody can give them a 10 hour project. So it's It's problematic that perspective. It works well, though if you are works better than than other companies, it for companies who are very skilled, focused, Um, so you know a good time where it's not quite as bad is a company that's like doing contracting anyway. That's that's where it makes a little more sense to dio uh, and with companies that place a little bit lower priority about algorithms, you really can't trust the persons performance. You can't rely too much on purpose algorithmic thinking if they were doing on there in time, because they might have relied on a whole lot of friends to do it. So, uh, but if you're gonna do it, that's where tends to work. Better for if you're gonna do it show interest first, Don't just assign this project off the bat. Make sure that your and you'll get better results. You better follow through anyway. If you've at least had some conversation with candidates about it, try to limit it where I I start object to homer projects more than more than get bigger and bigger and bigger. So set expectations should be a project that really can be accomplished in four hours. Eso I would start If you're doing this, make sure that you're gathering the data from candidates about how long it took them. Now, remember, can it's might very well not be honest here. They you know, if you expect to take four hours on, took 10 hours, they may not tell you that they may tell you four or five hours, but someday it is better than nothing. So that would make sure they're really cooking state on this, uh, look at more architecture of the project coding you maybe coding style. But over architecture, not so much performance on algorithms because too much cheating there make it clear accounts what you're looking for. Is this a basic like, can't you? Are you really looking at how good of a job did they dio? Or is it a basic screening? If you know what sort of things were you looking for? You looking for new handling edge cases? And, um, are you looking at the appearance of the application? If that doesn't matter, make you make that clear to them. So tell them what kind of things you're looking for. And one of the reasons I think often accompanies don't do this is that they haven't defined it themselves. And that's pretty frightening. So you should have defined these criteria with interviewers who are reviewing this. So I'd really defined. I'd really fried that information if you can, to candidates well and then avoid confusion with company work. So I know that companies, generally speaking, are not going to give cancer project as a way of getting like free labour out of them. I told that that's probably not a very good idea. Most first, almost can't. It's won't do a good enough job that you want to integrate it in to you would be best, you know, best case you'd be saving 10 hours if the Candid spent 10 hours on doing this. But then integrating all this other thing. I know that you're not gonna go try to get free labour out of Canada's, but I hear a lot of Countess who feel otherwise. If you give them a project that relates a little too much to something you done that you might actually want to integrate in, they can often walk away thinking that you were just kind of using them for free labor, especially if things don't go well. So you want to keep that in mind your project. Make sure it's it's very clear that's a toy project. We'll also add in is that if you are actually using this for some weird reason, you'd actually built this building into your application or use the code in any way. I'm not a lawyer, but I believe you do. Open up open door to a lot of legal issues as soon as you start doing that. If you're actually planning to use the work, the candid dust, so pick something you even picks on that's relevant ish, but just make it clear that's a toy project and not actual real work. Do they do code assessment tools? I'm much more in favor of when rolled out the right way. They're much shorter, so you're not burning the candidate so much. It's because they're for and because you can automate this, they are inherently automated. Uh, they can run through a lot more candidates. So with coding project with code assessment, talking about something like hacker Ankle, you give somebody a max to our tests that hopefully she will be taking them like an hour, giving them a lot of extra time. Its automated that can check, you know, can't check coding style but can check correctness on a contracted their actual ability to not just write an algorithm correctly, but also their ability to write something that's efficient. I've it says much order. So because grading can be automated, you're not relying on engineer to get around to finally read. You're looking at what they've done, so you can do it lama faster, and then you can Also, it's nice for recruiting perspective because now you can actually throw wider net out there. Look at candidates who may be didn't have great resonates, didn't look good on paper, but maybe they actually are, so it offers a lot more benefactor. I think to a company. It's also nice to do this because it can set expectations for on site interviews. So if you now you have to make sure that it's setting the right expectations, you asked Really easy questions. You're going toe set that expectation as well, and then the one thing that I really, really like. And part of why I often purged encourage companies to use coding assessment tools is that look, I do a lot of interview training, and a lot of new interviewers aren't great and went particular when you're trying to shift your hiring process in place, that interview with my feet in the wrong thing and the nice thing about a code assessment tools, if you set it up well, that's a consistent data point. That's not this interview, had a bad day or was a candid bad. It's a data point, ends consistent and that that's meaningful. And it gives you that sandy check of I wasn't so sure about. This interview is performance. If the interviewer do the right thing here, but they can also did poorly, so I feel pretty good saying they're not. They're not impressed. I want because it's in data point. Not to really useful. Some drawbacks, though, is there is cheating. It is absolutely a concern. There is. It's company, a hacker and cast plagiarism detection stuff and all the stuff in. Yeah, it'll kind of work. It'll check to make sure company cannons don't copy and paste their code, but you can't stop. You can't really do anything to stop somebody from working a team of three people. So there is cheating that that's going to very much reality. It also doesn't know checks. Efficiency checks. Correctness doesn't check coding style on bigger picture architecture sets a topic. It can also turn off senior candidates. Eso people who feel like they were treated like fresh grads who feel like they experience wasn't valued that can turn them off again. You know, I'm not dogmatic about things. I'm very open about the good things, the bad things. I think that assessment told tools are great when implemented well. But there are these drawbacks. One thing I'd say in response to the senior candid issue is, yes, it can. You should be doing these things if you're gonna be doing the same thing on site, in which case a lot of the senior Ken's who'd be turned off by an album test that's automated would are actually just fundamental turned off by album assessments and they'd be turned off on Can't on site to. So, yeah, it'll annoy some people, but a lot of people would get annoyed by anyway. You shouldn't really, You know, the idea of these tests is that you should be doing these Really, if you're gonna be doing the same thing on fight. So don't put in hacker rank and say, Well, now we checked off that album block. So now we don't have to do on site. That's very much not the case, because they could have cheated on various other things. So you're only doing these if it's replicates What? You're what you'd be doing on site anyway? A couple of ways implanting in. One thing you can dio is you could give them out to everybody very concerned Data point. That's also nice. You can also give them to just the may Be candid. So I've seen come from Some companies do that. They say, Look, Candace who they are very clearly not a fit. We just reject cans were very clearly fit, and we really think they have a very strong background. So you know somebody who who is, say, Ah, Google or Facebook engineer It's not that high port priority to give them this kind of assessment because in order to get it's not that of saying all Google or Facebook engineers are fantastic. There's gonna be weak engineers, any company. But they already a pass an equivalent assessment to get that Facebook or Google offer. So you know, it's less important to give them that. Yes, you could, but it s so why not use? Let the kind of a CAT scans who clearly, pretty clearly should do well on this? Just go ahead and skip this and give it to just people who are kind of that. Maybe maybe you aren't sure on that is one those places where allows you to cast much wider net. The other option, that is kind of cool that I've seen some kind of do companies. Dio is just kind of use it. Use it, condition some of these, but use those fast tracking. So a candid drops a resume, you eventually get around to looking at it. Hopefully But its accounts really excited. You merely pop back and say, Hey, we're gonna look at your resume. Assumes we can If you'd like to jump start the process, Here's a hacker and assessment you can take last sentences. Jump start the process can is we're really excited People who feel like they need to show that maybe they actually get some of this computer science stuff without this knowledge. Useful fact eso That's just different ways in planning it pros and cons of all of these. But I know all of these I'm pretty much thumbs up about no matter how you do it, no matter which way you do it. So it works great. It really any size of company other than maybe when you get to really, really, really small where you just don't have you, you don't really have any cans anyway. So certainly works best if you actually vow value out algorithms and problem and problem solving. Uh, so you works great for those situations. If you have a lot of Canada, that's where it's great as well as you has a lot of value. Looking at non traditional candidates, the people who maybe don't look great on paper, if you're gonna be doing these shows some, if possible, show some candidate interest first. So even just a call with recruiter will get you. Ah, higher rate of people who actually take this test. If you're the first thing they dio is, you get to get some things test. Not is a, like, fast tracking option, but like you to do this. Used to spend two hours taking this task before will even look at your resume that's going toe. Frustrate a lot of people, especially if they get rejected. Now. They spent two hours, and you just maybe they did well. And then you didn't like the rest Manute after them. That's not for them. Uh, but beware of cheating on the reason why I say this isn't a big deal, is that Look, what you're using this for is to build throughout candids more efficiently. You're not replacing the onset interviews you're not saying Well, they did well in this hour. Them test. So now we're gonna not do it on you, only using it to filter people out more officially. So people who cheat people who aren't good out rhythmically the harm that they're causing is you gave an on site interview you didn't need to give. So cheating is an issue. That's why I say Don't rely on it as you know, the only test of algorithms skills. But I don't think is a big deal, because you should really be doing this on site as well. Set clear expectations. I hear a lot of there's this Facebook group. I'm part of it, 25,000 developers or so on it. And a lot of people there saying, Hey, just how to take this Companies could assessment, tool. Anybody anybody take done that one before. So set expectations. Candid. So they're not reaching out to these Facebook groups or, worse, K or even worse, not knowing that, knowing that they can do that. So said annotations, give people an example, the kinds of questions they ask so that they don't, you know, get surprised by. And then I ask unique and challenging questions. If you're asking the same old boring questions and those you're not actually assessing really their problem solving skills, so asking things that are not super common and ask things that are actually challenging if you're just having them do a very, very basic coding challenge like no check of an array has duplicates. And just what do you really? I just don't think you're adding that much value there. So after that, actually acquire some thinking the perfect question. And this goes for on site interviews as well. If the questions I love or questions that have a correct and Sen my obvious answer that most people should be able to do is such that if you can't find a correct answer, that's a really, really big problem. Um, but also for the cans who are very strong, there was to think and get something even faster. That's the kind of question I love. Now, you know, this can. It was actually clearly really bad. They couldn't even get something fairly clear and obvious versus you know, they were okay, but they got some correct, but they weren't didn't quite just, you know, really, really smart. So that's kind of love diseases, and that's challenging has multiple kind of obvious solution as well as the less obvious, more often all solution and then 1 to 2 hours is the right ballpark. What I'd like to see with companies is an expectation that can is to generally be finishing it in an hour or so. But you're gonna give them two hours, just in case they get distracted, get confused about something. So, like that, um, if your code or you know that you can't always estimate how long something's gonna take, you get that one little bug and it takes you an extra 30 minutes of stupidest little thing. So you want you want tests that are that generally don't require that much time to sell it awfully mean set expectations with what you're looking for. Are you looking at coding style or not? You're not looking coding style. Make that clear to the candidate and don't look at coding style. You actually follow through on that promise. If you are looking coating sound. Make that clear the count as well, and that will help him kind of figure out what the right thing to do with. All right, so that's that's coating assessment. Homer projects any questions here? No. One thing we've tried inside that scramble combination of that and open and project and kind of like a panel interview where we have the kind of come in and they're working on a project that would take about 3 to 4 hours. It's kind of open ended. They get breaks in between, and it's something that it starts out with two or three engineers that kind of introduce themselves. And then they leave the room. They come in and out throughout the process and try to just help with this collaborative project. And then at the end, they present what they came up with, and we can just walk through it. Have you seen them? For what? Your thoughts? Yeah, so that's that's getting into kinda pair programming model and anything that those could be very effective, I think there Ah, great way of looking at someone's coding skills. I don't think you're gonna get as great of a read unlike here. Is this really hard alcohol problem? Here's how they and let me look at how they sold it. But I think for looking at someone's coding skills, it's actually great thing and works. Gonna have a guest speaker, uh, who's actually talk about pair programming at their company, but I think I'm generally pretty in favor of it. Uh, yeah. Thanks. Um, the take home assignment for then, when a lot of candidates apply, it's also a we we only administered to people who show a lot off interest. And also the other thing is, it's also a good signal off. How interested are they really in? Like if they complete that exercise that shows that they're really committed to spending that hour or two? Do you see it that way as well? Or do you think it's It's just unnecessary and, like we should fast track people who are like going from like companies where we know they have control. Rigorous process. So I do agree that someone's willingness do homework project is a reflection of their interest. Uh, but it's not really a reaction, their interest in your company or their passion for your problem. It can often be a reflection of the fact that we desperately need a job. So you're not a so looking what you think you do. Um, it's also, I think, a little bit unfair to say, I don't know if we want to hire this person because based on our name alone and the vague description what we do, they're not that interested like part of your job is to get them interested. And so I don't I don't love it from that perspective. It is a way of doesn't show interest. It just shows you desperation for that job, not necessarily a passion for that problem. If you want to look a passionate the problem, that's where you don't talk to them about it, that kind of stuff, rather than giving them making them spend five hours to show that passion.