Hehe, I talked about how I feel that a community of testers is the future, vs regular test teams, and of course there will never be agreement.
I didn’t realize this was such a sensitive subject, people are jumping all over it. So I feel the need to add some detail myself. I’m also gonna answer the comments I see so far.
I’m going to ignore all the blog posts that assume I’m going with the Huxley theory that enough monkeys typing can write Shakespeare. People that head off in that direction clearly never read my comments at the bottom of this posting.
“there will still be jobs for testing coordinators and test managers as the act of testing will never go away”
I never said that MOB testing was the solution, obviously it must be managed, hence my statement above. It’s annoying to read posts popping up assuming I think mob-testing is the future. Duh!
Regarding specific areas (security, balancing, testing, item costs, interface etc.) of course you need to focus individuals on each area. (Management again.) But in my experience you find in large groups people that get very specific. Like you will get one guy that just cares about Daggers. So much so he reverse-engineers the dagger system then compares it to multiple other games and presents a very compelling case. Maybe you’re lucky and have the dagger guy on your test team, or maybe not. (Which way gives you a better chance?)
On the other hand, I still stand by my comments, it’s a numbers game. If I’m looking for a random crash bug (the worst kind.) I can take in my 20 guys (hired from the community) OR I can take in 100,000 guys (from the community), neither will do a great job without a plan and oversight (so assume we have that), but I’d bet the 100,000 guys hammering away find it first.
Regarding getting testers to report bugs, you need to make it easy, including screen captures, they need to be able to see what others have posted to avoid repetition, you also need to reward the ones that deliver the best feedback. (So ignore the notion that it’s free. You have to pay to manage, support and offer incentives.)
Another thing you get from the large groups is you can survey them as well, to get large considered opinions. (Like thousands of opinions (the consumer), not the 5 paid guys that played that character.) If only 10% take the survey, you get 10,000 opinions, that’s tough for a programmer to argue against.
One controversial statement I’m also happy to make is that many of the test guys out there are just too good. They get the job because they kick a game’s ass. That sounds great, but it also means they tend to sku the difficulty higher than regular consumers can handle. I’ve seen it happen many times.
Regarding exposure of the game. Yes, the answer is not to put all 100,000 on the game day one. The trick is to roll it out. So as the game gets better more people see it. You are also turning on systems as the testing continues, so they have something to look forward to every new build.
Anyway, these are just my opinions. Some will agree, some will think I’m nuts (true) which is fine, but please don’t start posting that I’m advocating un-managed mob-testing as that’s complete nonsense.
The original posting…
http://www.fileplanet.com/fileblog/archives/2007/03/entry_232.shtml
Before I say anything, I do agree testing is an art form, testers are cool people, and many testers go on to be great developers. They have also made great friends over the years.
So this is not about the testers. It's a numbers game.
(1) To say that paid testers are better than everyone in the community would be dumb. (Considering we hire most of them from the community.)
(2) There's no degree course in testing that I'm aware of? Meaning a professional tester is no more qualified (on paper) than a community tester? (Other than they get paid.)
(3) Never just assume you are the best at anything.
(4) 150,000 people testing, kills 20 people testing, I don't care how you slice it. Interestingly testing needs really bad players too, yet I've never seen a test team hire some? That's the guy that jumps at a locked door 100 times because he has no idea what he's doing, and uncovers some great bugs testers don't find.
(5) Self-testing will also be a major future for games. Where the game plays itself, trying everything everywhere (to the extent feasible by the endless combinations generated) again every point on every surface will get tested for penetrations etc. A job testers can’t do accurately and it would be mind numbing to do this all day every day for months.
(6) Community testers contain hackers (rare on professional teams), these guys try every tool, every exploit on your engine and look for vulnerabilities. Again you don’t just get the one that might be on a professional team, you get dozens of them.
Yes people will still hire testers, but less and less over time. That said, there will still be jobs for testing coordinators and test managers as the act of testing will never go away.
If you think I’m crazy then tell me, but for me, I’m watching the industry changing rapidly from the inside, so my advice (which has helped many people get jobs in the past) is that becoming a student now, focusing (and becoming a Jedi) at a certain discipline, is a much better way into the business going forward, not testing.
dp
Posted by: Sam Kalman ![[TypeKey Profile Page]](http://www.dperry.com/mt-static/images/nav-commenters.gif)
at March 23, 2007 10:41 AM
150K testers is better than 20 if you get some info from those 150K testers, whether automatically or through them reporting. I've been in so many betas I can't remember and in every one of them there's people that get in just to play the game before it's released. They never report a single bug. Unless you're letting those people in to stress-test, they're just dead weight. :(
Posted by: Machaira at March 23, 2007 6:21 AM
The main problem I see with that kind of massive testing is how you will be sure to keep everything under wraps. Game Testers don't sign NDAs for nothing : they see things way before anyone else outside the company, and allowing 100 000 + players to test your game at once would make it really difficult to control leaks and the like. I don't think the industry is ready to take this risk because of the fierce competition. Maybe everyone could sign NDAs, I dunno. It asks for serious planning.
Posted by: Kudgel at March 23, 2007 4:42 AM
I so agree, it seems now days that the level of players has risen, and due to this bigger closed beta's aquire you a better chance at some of the best testers that care about games out there. However i wonder how long before the game industries start to compile and share information in a database (idea i had) that flags players as good tester, poor tester, hacking tester, hacking cheater and so on. I would actually love to see one company open its doors to create this database and offer it to all gaming company's. A system where you can identify possible threat players (cheaters who cheat to cheat) quickly and watch them from day one of entry into your game. This would require the players however to sign up with the system and have a Unique User name or ID (game id) that would be required by all games as a additional login when gaming.
IE my Game Id might be bob324 while my User name and password to your game is Joe677 password .... Thus the login would go from just a user name and password setup to a .... Game Id, User name, and password setup.
I think if IPsec7 ever takes off every person in the world will have their own unique IP that is used to log into the internet no matter who is paying for the connection. And then we might see this as a added feature into games, requiring your IP to log in, thus allowing gaming companys to sinstantly ban individuals who hack games to ruin them. One day maybe.
Posted by: taalon927 at March 23, 2007 12:00 AM
Thanks for signing in, . Now you can comment. (sign out)
(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)All Original Content © 2000–2006 David Perry — All Rights Reserved
A copy of our Privacy Policy is available here
From what I've read about your opinions on testing so far, I feel that you're categorizing many different types of testing together and calling that lump sum "game testing". With a primary testing base of consumers & gamers, you'll be limiting the scope of testing to a strong playtesting base. Sure, there will be people who try and find exploits, cheats, and hacks, but that's still under the play testing umbrella with maybe a touch of security testing.
Having a great big mishmash of all these people doing a lot of overlapping testing may be cheaper, but it is not more efficient. A smaller group of testers who explicitly focus on each of these areas and continually build up knowledge from experience testing the game will be more beneficial from the perspectives of accountability, coverage, and support.
I think the paradigm that you're focusing on is an elaborate beta test. No amount of gamers who are "testing" from the perspective of "how much fun am I having?" or automation code will be equivalent to a knowledgeable tester who has been with the project as it has evolved.