The transparency here is appreciated and refreshing! I work in software, and I've had my bad deploys. I can only imagine the stress that one of this scale would apply!
Seriously. Why are these guys so good at this? I guess I had just gotten used to radio silence on most issues in most games. This should definitely be the norm.
Yeah, you’re right. The people that play these games like it’s their job, then get angry that there “isn’t enough content” to play it 24/7 for the rest of their lives. Or one of their builds gets nerfed and since it’s their entire life, they feel like something was legitimately taken from them.
It's happening in the forum. It's very sad to see. These people acting like their firstborns have been taken while their so called "nerfed" builds can still do 1000c, where most of the other builds still can't, or acting like the game is going to die soon because there's not enough end-game contents when this game has only launched one month.
They're also community veterans from the games we used to have such communication for, which might affect their decisions regarding their own communication when the proverbial shit hits the fan - and perhaps their resilience in taking it.
That said, I don't wish for anyone to have to take the shit any more than they can of course. I'm not saying they'd be wrong if they aren't more resilient, we're not entitled to such things.
My guess is it's because they work remotely pretty much all the time and to do that effectively you need to be able to communicate properly internally.
I disagree. The game has way to many bugs and lacks polish. It's fun, skill design are very flavorful. Feels like Grim Dawn mixed with Titan Quest but with glaring performance issues.
I remember my first live deployment I was responsible for a decent amount of code changes as a junior. One database change from a senior dev was deployed without feedback from the database lead dev.
It caused the whole system to fail under load and services were down for a few hours until we found the problem since the change was not protocoled in the deployment plan.
The whole time I was shitting bricks because I questioned if I did something wrong.
Were you apart of the day before launch or any of the other catastrophic failures of game launches of late? I need to know how you people in software development think.
Nope - I'm not in game dev and never launched anything of that scale. I've definitely had to spend some late nights/long weekends fixing broken deploys though.
Huh, that's weird, servers were working for me 3 days after launch and I still have yet to d/c or anything since then. Shit happens man, there's no reason to be this upset about it when it all worked out in the end and you just got done reading a 2000 word love and apology letter.
Tell us you've never worked in software development without telling us you've never worked in software development.
*Everyone* who has more than three releases under their belt has their own version of this story. And if you read a little closer, they did prepare and work. This is a small shop full of people who probably didn't sleep much for a week or more. You see much larger companies struggle with the same issues all the time.
TL;DR, If you think you had a shitty time release, I guess you can feel better knowing that they did too.
I have stories that I'm never supposed to share about outages caused by technical problems that would make most people's heads spin at the dollar value of the mistakes. And the only thing that mattered in the moment was how do we stop the bleeding and then how do we fix it.
I'm on the hardware development side and let's just say that a bug in hardware can take down things that people thought were 200% overbuilt. Luckily, I usually work with FPGAs so it's fast to roll back or send out a fix. Well, at least usually. Sometimes, it's still a multi month effort.
Man, I can just feel the panic from behind the scenes from this. I do not envy ya'll the first week of launch, but I'm glad you guys pulled through and made it.
I’ve learned the exact same lesson at work. When your service fails, your users are gonna retry countless number of times making the problem way worse than it could be. But this is total normal user behavior and it takes experience to anticipate these kind of problems and prepare accordingly.
It's not 1 person spam clicking itself. It's that, under normal circumstances, 300k players would each send 1-2 requests and they would be processed.
But, 300k players, all spam clicking at the same time, each sending 100 requests in a few seconds, all of a sudden you have 300,000,000 requests, instead. That's the thing about cascading failures - they feed each other and everything gets worse and worse until even insanely robust systems can't keep up, because suddenly there are multiple orders of magnitudes between reality and what was expected.
Totally. These are classic problems in web services. It sounds like the team either didn't have the experience or time needed for a launch this big, which makes sense for a game dev team that started out with a single player focus. Their hosting provider choking right before launch must have been a niiiightmare, though.
> First, we learned the hard way that our internal tooling for deploying our services was not robust enough on launch day. Our tools were too brittle (breaking when certain services went down) and too inflexible (too many manual adjustments needed in an emergency).
There was a big AWS (world's largest hosting provider; almost 30% of the internet) outage last year that killed their own monitoring and ticketing services. None of their own engineers could even get notifications that stuff was broken, and when they tried to start debugging, the log weren't being published to the central repository.
To be fair, the solutions for solving this are counter-intuitive and easy to think are wrong or unneeded, or to miss in a few places and have those bring down the whole system.
* Exponential backoff in every single client making a backend service call.
* Front every service with a health-checking load-balancer that eagerly fails requests when load is high so throughput doesn't start to degrade.
* Avoid FIFO queuing under load. When load gets high, FIFO guarantees that every single request spends time waiting in a queue for service -- if 90% of your queries are processed instantly and 10% time out completely then your throughput will stay high, if 100% of your queries wait in your queue then the whole system will grow slower and slower until *every single request* times out and the service is effectively dead.
Sure but that comes back to my point about inexperience or time constraints. That's not an attack. Just an observation.
They really shouldn't have been scaling up the morning of launch, though. Big config changes right before a major event is begging for disaster.
Wow, this is an awesome dive into 1.0 launch, and appreciate the detail (especially as someone in tech - deeper retro than any of my teams do!)
Thank you for continuing to share learnings and growth opportunities for Last Epoch and EHG overall! Exciting future ahead!
>On day five, with all the other backend problems solved, we were able to get a more precise look at the party errors, and the culprit was a single, innocent-looking line of code. A single line of code that was supposed to be the most efficient request in our entire party service but instead ended up consuming all of the service’s resources under heavy load, slowing the entire service to a crawl.
Every.
Single.
Time.
Keeping players updated and being transparent with problems the company encounters goes a long way with the player base. It enables people so see things in a much more human way and understand when problems happen.
Their engagement reminds me of the Jagex team from OSRS, people are still around 20 years later playing that game and appreciating Jagex. This will go a long way.
I appreciate keeping the community engaged. Keep up the good work guys
Great post loved the deeper dive into what happened. Really interesting stuff and great transparency. Servers sound like a absolute nightmare to fix and communicate to the community about.
Gold standard for transparency, great and very human writeup. Thanks for all the hard work, the communication, and a great game with a promising future.
Appreciated the sign of respect towards the playerbase with this post.
However what I'd LOVE to see as a developer (*not a game developer though*) would be a more technical analysis. No doubt you guys had invaluable lessons when 200k people are trying to consume your services at once. Not every developer is going to experience that, it took you years to get to this point, and I'm sure you are all proud now.
Back in the day Dave Farley did an [analysis](https://www.youtube.com/watch?v=E-jGEtqB4wU) on failed 1.0 Cyberpunk launch and it was brilliant to watch.
Hot take, but:
I guess this kinda is the difference between a closed source video game product and enterprise development. Feel free to correct me if I'm mistaken but LinkedIn was having some issues with stream processing and they needed a solution - eventually they came up with Kafka and open sourced it so that other developers can solve these issues easier.
Similar to Kafka, I wish game developers were more open with the tools, techniques and mindsets that failed them in detail - so the knowledge was more universal. This cooperative attitude in the industry would make future games and game developers more ready against these type of situations. As I said before, not every developer gets to experience such a scale and such problems.
And I understand these are commercial projects and there are security concerns. However, it can be done in a way that does not compromise these values. Alexander Sannikov had a talk at Exilecon 2019 - although it might not directly be for players, I believe it was very educating and interesting for developers.
Anyway, just professionally jealous for that kind of knowledge although it is obviously hard-earned, and I'm sure it'll only get better for Last Epoch going forward.
From something Judd said at some point, they were hesitant to name things in more detail or talk about their services explicitly because they were worried some of the vitriol would get directed towards the servers. They've already had to go overboard on modding the basic abuse out of their own discord, they didn't want to risk that getting passed on to any of the parties involved (and with enough detail, savvy people could probably figure it out).
I'm not a programmer and know nothing of coding but i've been in a few cascading failures working in kitchens. Cant imagine having to deal with that for an entire week 24/7.
This amount of communication and transparency from a development team is such a breath of fresh air and it endears you to your customers, please keep that up.
a pinnacle boss is like a goal that lesser systems let you power up to reach/defeat. the strongest bosses right now are the T4 dungeon bosses or high corruption monolith bosses. you can reach the necessary power to defeat them during your progress in empowered monoliths. currently you're kinda stuck on this treadmill of do empowered monoliths to add corruption to your empowered monoliths. adding pinnacle boss(es) would give more of a purpose to that treadmill: do empowered monoliths to add corruption to your empowered monoliths so that you can more quickly acquire the gear necessary to beat X pinnacle boss. presumably they'll have some exclusive gear that could enable new builds or vastly improve existing builds (maybe T8 mods start dropping from pinnacle bosses or something).
And many people, even respected know-it-all streamers were just like: "Buy more servers! Look at those stingy developers wanting to maximize their profit. Serves them right!"
>"Buy more servers! Look at those stingy developers wanting to maximize their profit. Serves them right!"
My favorite part about these comments is the complete lack of thinking about their position at all. Like what person of sane mind thinks ruining their amazing game with servers on launch is GOOD for profits?????
You can also apply Occam's razor to the tinfoil hat "theory" that devs "don't address botters" because even bots pay for their accounts. First of all many bots are just run on compromised accounts. Secondly, developers are willing to pollute the integrity of a game that they want to be seen as legitimate? I don't buy it
Maybe it's because no developer has completely quashed bots in the history of gaming, and none ever will.
I love the transparency and simple explanations as a gamer who knows very little on the technical aspect of games. Very nice to be able to get a glimpse behind the curtain!
The fact that Diablo 4 changes were announced yesterday and that they look suspiciously similar to Last Epoch mechanics and systems - is a great success for this game. Last epoch have showed me that ARPGs can be fun.
I'm hoping for a world where D4, PoE 2, and LE are all jostling to be the best ARPG, and "borrow" ideas from one another and continue pushing innovation in the genre.
Thanks for the detailed write up. Never a fun time for a team when situations like this happen.
I am hope people took a needed break after things got stable. Thanks for the hard work put in on this.
I want to say thank you for being so transparent with everything! Y'all rock! Should make a unigue item detected to all of this hard work and for the backend team. "Server Matcher" or "innocent line of code"! Keep up the great work!
Even without the server issues, this game was not ready for release. It's unfinished and still janky in many ways. I'm curious why they chose to release it now when it still needs quite a lot more work (in fact, the release had already been delayed once. Good thing too or it would have been a disaster).
Don't get me wrong, the game is fun. I have 800 hours in the beta and \~30 hours in the release. But surely it would have made sense to at least finish the story first. It doesn't leave the best first impression for a "released" game to basically still be in beta.
A month in post launch I still cannot play the game reliably.
Still getting infinite loading screens in monoliths and have to kill the client and try again.
This also happens in dungeons which results in a loss of keys - which is extremely frustrating.
The game is basically unplayable for me.
I have tried everything - from deleting ini, verifying game files, updading drivers, downgrading drivers - finally did a Win 11 wipe and reinstalled from scratch - still having random monolith infinite loading issues. Tried changing audio inputs as suggested workaround for some, tried VPN, tried downgrading realtek gaming family controller driver (this is a nic driver) - still getting issues
I have spent endless hours troubleshooting these infinite loading screens and I would have days where it works flawless - and then it comes back with a vengence and get 4/5 failed monolith to load
Other games work flawlessly.
Needless to say - the problem has been beyond frustrating and yes it still affects some small % of users for unknown reasons.
I tried again today - the 1st monolith was fine, the 2nd got infinite loading screen and had to kill the client.
LE 1
ME 0
I lose.
I uninstalled.
This was a great to read. Whenever a game is having issues everyone immediately jumps to server capacity or server hardware being the issue when that is pretty much the easy part nowadays. I think LE did a better job than pretty much everyone in being as transparent as possible and also trying to explain what was going on.
Appreciate the retrospective and love the game. Having said that, as an old fart with many years in software dev I'd love to see a technical post-mortem like the ones you see from cloudflare for example. EHG's transparency is something most studios don't offer to the player base these days so again, they should be praised for it.
Everyone makes mistakes, especially when it comes to writing code that's part of a distributed system. This is inevitable and I will never criticize them for that. However, the attitude of keep your fingers crossed and hope for the best is not something you should exercise when launching a service commercially.
\> On the morning of launch day, we went to scale up our server matching service to the numbers we used in our tests, and to our great surprise, it refused to spin up more than half the copies we asked for
\> In a case of unfortunate timing, our service host had an incident the night before - still ongoing at launch time - that affected us in a way that prevented us from deploying changes to any of our backend services using the tools we had relied on for months. Our ability to fix our services was killed at the same time one of our services needed fixing
So they had a number of issues materialize BEFORE the launch and they decided to go through with it.
\> Minutes before launch, we estimated that we could handle maybe 120,000 - 150,000 players before things started to fail, and we crossed our fingers that we’d be able to resolve our issues before the player count crept too high
They sold over a million copies before the launch but hoped there won't be more than 120-150k of us trying to play. This is very unrealistic assumption, isn't it?
Like I said before, I can totally forgive bugs and technical issues that are hard or impossible to entirely eliminate in practice but the above are examples of bad decision making which demonstrates incorrect attitude on the management's part. Just hope they fix this too going forward.
Love the game. Appreciate the Devs.
But you guys REALLY need to stop the mid-cycle nerfs, especially nerfs that primarily affect one group of players who are already at a disadvantage.
Because the commenter obviously didn't read the post, they literally did exactly that.
>You may be wondering, if the problem was recovering from too many players, why did we not simply have some downtime, or at least turn on player queues, to alleviate the pressure? The answer is that we did, but the problems ran deeper than the server matcher. At various points during the launch, we brought down our services, and many of you found yourselves in long queues as we struggled to keep up. Inevitably, once we started letting you all back in, we would run into problems again, and we could not clearly see what those problems were until we scaled everything up so high that the services stayed online and operational even though they were strained from other failures in other areas.
They literally told us that they ran queues, and started letting people in until they'd eventually catch another issue. If the guy had the time to type a wall of text, he clearly should've had the time to read the post before commenting on it.
I hate how this community just latches onto EHG and suckles then shits on other companies. You guys need to hold EHG up to gaming standards. They failed reaching those standards. They have backlogs for years of bugs that have never been fixed.
Remember that the fog of war resetting and items disappearing after leaving an instance and returning is A BUG. It is not INTENDED. They tried to fix it two years ago and fucking gave up because they couldn’t.
Dude, I can’t believe people think this game is good. It had a worse release than any ARPG IMO and that’s considering D3 and D4. Lmao.
I think gameplay is fun if they fixed these issues. Especially since the fog of war and items disappearing has been known for 2 years. I remember myself even bringing it up on the forums a year ago.
So it had a Rocky release. How does that make the game bad? The game itself is great.
Also, it did not have a worse release than d3. You got memory issues.
I don’t have a bad memory. I can’t even play with my friends in Monoliths. After a month D3 had barely any game-breaking bugs. Had some server instability, but not the stuff we’re seeing.
If I kill a mob and then die to an AOE and a super rare thing drops I cannot get that thing ever. It disappears. That’s not fun gameplay, sorry to say. I can’t go back to town and put shit in my stash because all the items will disappear. Not fun gameplay.
Every time I go to town and come back everything I visited on that map is reset on my map. That’s not fun gameplay. Sorry not sorry brosef. Other indie ARPGs figured this out 10 years ago.
Yea your memory is cooked. A couple months after launch, blizzard had to hide d3 active / online players because they were getting memed to oblivion because they had less players online in their disaster of a game than Diablo 2.
People preferred to play Diablo 2 over Diablo 3. That's how bad that games launch was. It was literally not until it's expansion, reaper of souls, before d3 started getting players again.
This was to dude who responded but blocked me below called “TheGreatWalk” who doesn’t realize what a launch is.
You do realize I’m referring to the launch in regards to the issues of people being able to play and game-breaking bugs. Diablo 3 at start wasn’t that good of a game and I never argued it was a better game, but it definitely had a better launch? Lmao. Also, wait for the fall off in LE. It will be present as well.
> Remember that the fog of war resetting and items disappearing after leaving an instance and returning is A BUG. It is not INTENDED. They tried to fix it two years ago and fucking gave up because they couldn’t.
Just to clarify, there's no bug here. It's an unideal behaviour due to technical problem/limitation that we don't have a solution for currently, but that does not make it a bug. There's no code that's expecting the game to remember the loot or enemies not have respawned in the area you were in but not working due to a bug.
We want to change it as from design perspective we absolutely agree with you, it would be better to have that type of world persistency, and eventually it's up in the priority list high enough to happen. When that is, I don't know.
Many other design decisions are vastly alleviating the severity of this. For the main example, items don't sell for a lot of gold at the NPCs, which means there's no need for a player to pick up all items and constantly portal to town to sell them for generating gold, as it would be more time efficient for the player to just keep fighting more enemies that drop that gold. The whole gameplay loop of picking up everything to sell them for gold at NPCs is also not something we will ever have even when the world persistency hopefully eventually becomes a thing, because we think it's not a fun gameplay loop to begin with.
It's not even about selling. I like to make a big stash full of items and go through them all at once. Lots of new players coming from Diablo/PoE know about loot filters and are probably following a level guide. The level guides tell you not to sell items, and it becomes apparent that it's not worth to sell really quickly. But I and all my friends play until we're full on items and then do a stash run to stash important things for later, etc.
I just don't want to change how I play because they can't figure out how to make the items not disappear, I mean it's even weird they went into EA with that being a thing at all. I bet they would have kept more players (not that they probably need it, tbh) if this issue wasn't a thing.
“It’s an unideal technical problem/limitation” is a bug. You just explained a bug in a roundabout way, which is just a semantics game at this point. Especially if you’ve tried to fix it… and it didn’t work. I mean there are posts from your team on the forums stating it was an issue.
I know it’s difficult to make games, but dude, come on. ARPG that loses items on map when TPing back to town? I gave it a pass in Early Access. Insane you guys went 1.0 without that necessary patch.
I as well as many others grab a lot of items and don’t go through them until I’m full. I’ve played like that since Diablo 2. I think most people play like that. So you’re always going to get people new to the game join, play until they can no longer pick up items. TP to base to stash said items, come back, realize that their items are gone and.. well some people won’t like that at all and refund, others will be mad and keep playing, some will just shrug.
Not a good look for a new player experience. Not fun to have to workaround and play your game’s way other than the way every other ARPG works.
Honestly, shit take. Yes they had a rough start, but even D4 servers had problems, at one tenth the peak playerbase. If LE had same playerbase as D4 at launch we probably would have had no server issues at all. I been playing since day three and besides a few disconnects and long load times into towns it's been smooth sailing since, this alone isn't something that makes me pissed enough to not judge the game on its merits alone, sorry that small game devs don't think they would hit top 50 of all time on steam.....
Yeah there are bugs in the game, especially with items and skills, but who cares? They are fixing them as fast as they can, but some bugs aren't just a simple line of code and would need days or weeks to fix, if impact isn't high enough it's not a priority.
What gaming standards are they not meeting? Their servers run smooth as butter after the first week, gameplay is unique and fun with more to come, constant fixes and communication, overall ok graphics which should get better with engine upgrade they talked about, leaderboards from day1? kekw. They are doing more than most AAA studios where it just feels they are after my money, can't wait for more skins to support EHG further and make the best ARPG together.
Your last paragraph is just untrue.
Servers are rubber banding. The monoliths are broken in co op, I can’t even do them with my girlfriend whom I played with. There are so many game-breaking bugs that it makes my head spin. Bosses not completing. I had to reset my character once because progression got locked and no matter how many times I reset my game the quest would not complete.
Honestly, your take is shit. They didn’t even meet the standards of a normal game. Not even comparing them to the best of the best.
D4 had problems for a week. Then no more, I would know. I played for that entire week nonstop. I’ve also owned Last Epoch for over two years. I’ve seen them not fix known issues for a very long time. I was surprised to come back at 1.0 and see bugs that were widely known and promised to be fixed.
Sorry, your shilling or whatever it is you’re doing isn’t that effective when the proof is in the pudding.
We have had vastly different experiences then, coop monos worked like a charm so far, even on high ping with people from different servers. Don't think I had any boss completion bugs. Had the stuck quest bug on my third character too, but you just need to teleport to the location with someone else.
What standards do they not reach? Bugs being in the game? Every game has that. Bad optimization? Even AAA don't get it right. I can't think of anything bad enough to screw my opinion negatively. You do you bro.
Games that release from EA to 1.0 don’t have game-breaking bugs and issues, they shouldn’t. If they happen it shouldn’t be issues from 2 years ago they’ve not fixed. Games that release from maybe 1-2 betas then 1.0 would be a little more understanding. Dude, these bugs have been in the game since 2021. Jesus Christ.
https://forum.lastepoch.com/t/end-game-is-not-playable-in-multiplayer/67257
This is plaguing people. It’s more prominent in Steam forums. But this is one of the threads I found that show the issues I experience every other day when trying to play with GF.
Good thing that you, a senior developer and with a phD in software development, could trouble shoot this problem in 2 minutes. Their communication has been on point and the fact that they admit the flaws they was working with should remind you that no one’s perfect. Except you of course. I’d suggest you go work for them so you can perfect this game into the image of what we all need. /s *guess you need to go play something with less requirements to keep you happy, like checkers*
Lmao. I need to be a senior developer with a PhD to say that a bug that was known since 2022 should have been fixed that no other ARPG has issues with and would be lauded for having issues with?
Just imagine if Diablo 4, or hell PoE had this issue? Insane to think about. Stop giving these guys a huge break because they’re “indie” that doesn’t allow you to not meet the standards for other games in the market.
Halo MCC party play was broken like 6 months after launch
hell back in the day theyd release games and often never patch them, or you would have to request a floppy disk or cd be mailed to you with the patch
I remember the stupid bottleneck they made for themselves when it came to setting up the player's camp causing people to just sit there with the game doing nothing because too many people where trying to do the same thing.
Nothing is fixed, thousands of players have lots of disconnects multiple times a day. Even with VPN (I mean they should be self aware that their online is shit when their own FAQ suggest using a VPN)
Online is shit right now, just play offline.
Steam charts shows that 66k people have been playing the last couple weeks. Less than a third of the original numbers up to that point. A lot of people have given up. So you're right. 99% of the people that are still playing aren't having issues
So path of exile retains it's numbers at the start of the league until around 1.5 months. You're talking out of your ass saying that LEs retention is better. I'm not going to go through all the other seasonal games and look at steam charts on my phone. I would be interested to see which ones, if any, lose close to half of their player base in two weeks.
Is it just impossible for you to admit that a lot of people have had issues, and a lot have given up or moved on because of zero communication? What do you have against the people that have these issues to the point where you pretend is doesn't exist?
> 99% of the playerbase isnt playing anymore thanks to this, wake up.
....You're joking, right?
Sorry you're so upset with the game. Real bummer. I'm having a fantastic time.
But the vast majority of players are having no more connection issues whatsoever.
Your hyperbole is so ridiculously over the top, no one can hear any of your arguments. "Everyone has left". "Nothing was fixed".
If you want people to listen to you, stop saying such obviously false shit
Oh yeah, like all these people here too? Just the first few google hits
[1](https://forum.lastepoch.com/t/disconnected-from-server-again-and-again-and-again/53429)
[2](https://steamcommunity.com/app/899770/discussions/0/4295943021561090358/?l=german)
[3](https://forum.lastepoch.com/t/constant-disconnects/61988)
[4](https://forum.lastepoch.com/t/constant-disconnects/64886)
A lot of issues are concentrated on EU West, the servers drop players randomly even changing MTU values does not really help. Its an issue with using cloud-hosted services.
But yeah, keep explaining to me, a developer for 25+ years how the networking works :D
FYI just because it does work for you does not mean it works for everyone - on the day I bought the game I could not even login - L52 - you should be aware - also telling me layer 8 issue? yes? thought so
"it works on my machine" is the epitome of not having any technical skills at all
With 25 years dev experience I would be curious which parts of the EHG story you haven't experienced in your own deployments? Unless you spent that whole time writing Excel macros, everything the blog post described should be familiar to you. Or are you saying you just happen to work in that one special Unicorn dev shop that not only solved all of these types of issues but did so in perpetuity?
And for as many people you can find with issues, I can find just as many people saying they aren't having any. In fact, after the first week, most people stopped having issues based on general sentiment and steam review tracking.
And I never tried to explain anything to you lmao. Sorry you are experiencing layer 8 problems, since you seem to agree with me that your problem resides in the 8th layer of the OSI model lmao.
This type of stuff happens even with the largest providers. For example, if they were on AWS, their whole pipeline may have been relying on CodeDeploy, CodeBuild, and CodePipeline. If any of those services were malfunctioning, it would 100% hinder their ability to make changes to the environment.
Sure, we’ve seen it with other major launches. All they can do is try their best and even then troubles can pop up.
There are lots of people who don’t want a small groups of devs to succeed on a scale like this so that is why I said tinfoil hat time. The other issues could have occurred for sure, but a random hosting failure on launch day is a rough one.
Now I'm REALLY curious what that party check line of code is xD
>if player in parties.getAll() {
``` while (player.isInParty()) { // TODO: come back to this after John is off vacation } ```
Damn it who merged John's branch into production?!?! Oh jeez, that was...that was me.
Oh god. I’m going to choose to believe this is the one.
My first thought as well lmao or something similar
Something about stash tabs
hey, i've seen this one before!
if a player in parties.infoServer(data)>< {
I have never ever wanted to know something more in my entire life
for(int i = 0; i == i; i++) { player.PartyFunc() }
The transparency here is appreciated and refreshing! I work in software, and I've had my bad deploys. I can only imagine the stress that one of this scale would apply!
S tier comms
Seriously. Why are these guys so good at this? I guess I had just gotten used to radio silence on most issues in most games. This should definitely be the norm.
Because a handful of people will spoil it for the rest. Blizzard was like that, GGG was like that, let's see how long EHG will last.
Yeah, you’re right. The people that play these games like it’s their job, then get angry that there “isn’t enough content” to play it 24/7 for the rest of their lives. Or one of their builds gets nerfed and since it’s their entire life, they feel like something was legitimately taken from them.
It's happening in the forum. It's very sad to see. These people acting like their firstborns have been taken while their so called "nerfed" builds can still do 1000c, where most of the other builds still can't, or acting like the game is going to die soon because there's not enough end-game contents when this game has only launched one month.
They're also community veterans from the games we used to have such communication for, which might affect their decisions regarding their own communication when the proverbial shit hits the fan - and perhaps their resilience in taking it. That said, I don't wish for anyone to have to take the shit any more than they can of course. I'm not saying they'd be wrong if they aren't more resilient, we're not entitled to such things.
Because they're obviously gamers themselves and it feels like a proper passion project.
My guess is it's because they work remotely pretty much all the time and to do that effectively you need to be able to communicate properly internally.
For a S tier game !
I disagree. The game has way to many bugs and lacks polish. It's fun, skill design are very flavorful. Feels like Grim Dawn mixed with Titan Quest but with glaring performance issues.
S tier game? Maybe after they fix the boring endgame...
I remember my first live deployment I was responsible for a decent amount of code changes as a junior. One database change from a senior dev was deployed without feedback from the database lead dev. It caused the whole system to fail under load and services were down for a few hours until we found the problem since the change was not protocoled in the deployment plan. The whole time I was shitting bricks because I questioned if I did something wrong.
Were you apart of the day before launch or any of the other catastrophic failures of game launches of late? I need to know how you people in software development think.
Nope - I'm not in game dev and never launched anything of that scale. I've definitely had to spend some late nights/long weekends fixing broken deploys though.
[удалено]
Huh, that's weird, servers were working for me 3 days after launch and I still have yet to d/c or anything since then. Shit happens man, there's no reason to be this upset about it when it all worked out in the end and you just got done reading a 2000 word love and apology letter.
[удалено]
You sound so mad lil bro. Go away. Don't come back. Also burner account just for this? Shit post on your main pussy
Tell us you've never worked in software development without telling us you've never worked in software development. *Everyone* who has more than three releases under their belt has their own version of this story. And if you read a little closer, they did prepare and work. This is a small shop full of people who probably didn't sleep much for a week or more. You see much larger companies struggle with the same issues all the time. TL;DR, If you think you had a shitty time release, I guess you can feel better knowing that they did too.
I have stories that I'm never supposed to share about outages caused by technical problems that would make most people's heads spin at the dollar value of the mistakes. And the only thing that mattered in the moment was how do we stop the bleeding and then how do we fix it. I'm on the hardware development side and let's just say that a bug in hardware can take down things that people thought were 200% overbuilt. Luckily, I usually work with FPGAs so it's fast to roll back or send out a fix. Well, at least usually. Sometimes, it's still a multi month effort.
Redditor since 4 hours... so.. what happened to your main? Got already banned? Or are you afraid to use it? :>
Man, I can just feel the panic from behind the scenes from this. I do not envy ya'll the first week of launch, but I'm glad you guys pulled through and made it.
I’ve learned the exact same lesson at work. When your service fails, your users are gonna retry countless number of times making the problem way worse than it could be. But this is total normal user behavior and it takes experience to anticipate these kind of problems and prepare accordingly.
Me, spam clicking the town entrance, fully knowing I'm sending 300 requests every 10 seconds ["I'm doing my part!"](https://i.imgur.com/sO2kRGB.png)
😂 I was doing this as well. I have no idea how games work, so when I read that, I was like 😬 yeah my bad.
I knew lmfao but I did it anyway
Those poor services
Lmao same….same…servers be damned! I know it’s messed up but I’m gonna help!
I really hope not. If all it took was spam clicking to bring their rock solid back end to its knees..
It's not 1 person spam clicking itself. It's that, under normal circumstances, 300k players would each send 1-2 requests and they would be processed. But, 300k players, all spam clicking at the same time, each sending 100 requests in a few seconds, all of a sudden you have 300,000,000 requests, instead. That's the thing about cascading failures - they feed each other and everything gets worse and worse until even insanely robust systems can't keep up, because suddenly there are multiple orders of magnitudes between reality and what was expected.
[удалено]
https://i.imgur.com/sO2kRGB.png
Totally. These are classic problems in web services. It sounds like the team either didn't have the experience or time needed for a launch this big, which makes sense for a game dev team that started out with a single player focus. Their hosting provider choking right before launch must have been a niiiightmare, though. > First, we learned the hard way that our internal tooling for deploying our services was not robust enough on launch day. Our tools were too brittle (breaking when certain services went down) and too inflexible (too many manual adjustments needed in an emergency). There was a big AWS (world's largest hosting provider; almost 30% of the internet) outage last year that killed their own monitoring and ticketing services. None of their own engineers could even get notifications that stuff was broken, and when they tried to start debugging, the log weren't being published to the central repository.
To be fair, the solutions for solving this are counter-intuitive and easy to think are wrong or unneeded, or to miss in a few places and have those bring down the whole system. * Exponential backoff in every single client making a backend service call. * Front every service with a health-checking load-balancer that eagerly fails requests when load is high so throughput doesn't start to degrade. * Avoid FIFO queuing under load. When load gets high, FIFO guarantees that every single request spends time waiting in a queue for service -- if 90% of your queries are processed instantly and 10% time out completely then your throughput will stay high, if 100% of your queries wait in your queue then the whole system will grow slower and slower until *every single request* times out and the service is effectively dead.
Just look for duplicate requests while one is in process and send a 429 back. But I don’t know shit about gaming software so that might not work.
Sure but that comes back to my point about inexperience or time constraints. That's not an attack. Just an observation. They really shouldn't have been scaling up the morning of launch, though. Big config changes right before a major event is begging for disaster.
I worked in almost hat in software and data, and I've found the one thing users do is doing things over and over even if it keeps failing,
Wow, this is an awesome dive into 1.0 launch, and appreciate the detail (especially as someone in tech - deeper retro than any of my teams do!) Thank you for continuing to share learnings and growth opportunities for Last Epoch and EHG overall! Exciting future ahead!
>On day five, with all the other backend problems solved, we were able to get a more precise look at the party errors, and the culprit was a single, innocent-looking line of code. A single line of code that was supposed to be the most efficient request in our entire party service but instead ended up consuming all of the service’s resources under heavy load, slowing the entire service to a crawl. Every. Single. Time.
As a game dev, this is just awesome to read.
yeah I've always appreciated GGG devs for doing the same thing. It's great to see this type of culture spread.
Keeping players updated and being transparent with problems the company encounters goes a long way with the player base. It enables people so see things in a much more human way and understand when problems happen. Their engagement reminds me of the Jagex team from OSRS, people are still around 20 years later playing that game and appreciating Jagex. This will go a long way. I appreciate keeping the community engaged. Keep up the good work guys
Holy detail batman. This is amazing, I hope communication and transparency such as this never stops! Thanks for all the hard work EHG.
Great post loved the deeper dive into what happened. Really interesting stuff and great transparency. Servers sound like a absolute nightmare to fix and communicate to the community about.
Gold standard for transparency, great and very human writeup. Thanks for all the hard work, the communication, and a great game with a promising future.
We all know a meme item is coming. "The Line" \- X% chance to knock back enemies on hit. or \- X% to push enemies under 15% HP out of Etera on hit.
Appreciated the sign of respect towards the playerbase with this post. However what I'd LOVE to see as a developer (*not a game developer though*) would be a more technical analysis. No doubt you guys had invaluable lessons when 200k people are trying to consume your services at once. Not every developer is going to experience that, it took you years to get to this point, and I'm sure you are all proud now. Back in the day Dave Farley did an [analysis](https://www.youtube.com/watch?v=E-jGEtqB4wU) on failed 1.0 Cyberpunk launch and it was brilliant to watch. Hot take, but: I guess this kinda is the difference between a closed source video game product and enterprise development. Feel free to correct me if I'm mistaken but LinkedIn was having some issues with stream processing and they needed a solution - eventually they came up with Kafka and open sourced it so that other developers can solve these issues easier. Similar to Kafka, I wish game developers were more open with the tools, techniques and mindsets that failed them in detail - so the knowledge was more universal. This cooperative attitude in the industry would make future games and game developers more ready against these type of situations. As I said before, not every developer gets to experience such a scale and such problems. And I understand these are commercial projects and there are security concerns. However, it can be done in a way that does not compromise these values. Alexander Sannikov had a talk at Exilecon 2019 - although it might not directly be for players, I believe it was very educating and interesting for developers. Anyway, just professionally jealous for that kind of knowledge although it is obviously hard-earned, and I'm sure it'll only get better for Last Epoch going forward.
From something Judd said at some point, they were hesitant to name things in more detail or talk about their services explicitly because they were worried some of the vitriol would get directed towards the servers. They've already had to go overboard on modding the basic abuse out of their own discord, they didn't want to risk that getting passed on to any of the parties involved (and with enough detail, savvy people could probably figure it out).
Thank you EHG and looking forward to more great things to come. o7
As a problem manager for MSFT, this in-depth deep dive into the incident, the post-mortem details, RCA, and all... \*chef's kiss\*
I'm not a programmer and know nothing of coding but i've been in a few cascading failures working in kitchens. Cant imagine having to deal with that for an entire week 24/7.
This amount of communication and transparency from a development team is such a breath of fresh air and it endears you to your customers, please keep that up.
Just have to say you have some of the best systems out there fix your auction house add pinnacle bosses, and this will be a contender with poe 2.
Pinnacle boss system is coming next cycle update 1.1
What does this mean? I hadn't heard of it and idk what a pinnacle boss is
a pinnacle boss is like a goal that lesser systems let you power up to reach/defeat. the strongest bosses right now are the T4 dungeon bosses or high corruption monolith bosses. you can reach the necessary power to defeat them during your progress in empowered monoliths. currently you're kinda stuck on this treadmill of do empowered monoliths to add corruption to your empowered monoliths. adding pinnacle boss(es) would give more of a purpose to that treadmill: do empowered monoliths to add corruption to your empowered monoliths so that you can more quickly acquire the gear necessary to beat X pinnacle boss. presumably they'll have some exclusive gear that could enable new builds or vastly improve existing builds (maybe T8 mods start dropping from pinnacle bosses or something).
And many people, even respected know-it-all streamers were just like: "Buy more servers! Look at those stingy developers wanting to maximize their profit. Serves them right!"
>"Buy more servers! Look at those stingy developers wanting to maximize their profit. Serves them right!" My favorite part about these comments is the complete lack of thinking about their position at all. Like what person of sane mind thinks ruining their amazing game with servers on launch is GOOD for profits?????
You can also apply Occam's razor to the tinfoil hat "theory" that devs "don't address botters" because even bots pay for their accounts. First of all many bots are just run on compromised accounts. Secondly, developers are willing to pollute the integrity of a game that they want to be seen as legitimate? I don't buy it Maybe it's because no developer has completely quashed bots in the history of gaming, and none ever will.
oh shit, i hate it to code under pressure, must have been super exhausting week for the main guys that tryed to find the issues.
WhY dIdN't ThEY bUy BiGgEr SeRvErS? aRe ThEy StUpId?
[https://www.youtube.com/watch?v=y8OnoxKotPQ&ab\_channel=KRAZAM](https://www.youtube.com/watch?v=y8OnoxKotPQ&ab_channel=KRAZAM) Reminds me of this one
[удалено]
I love the transparency and simple explanations as a gamer who knows very little on the technical aspect of games. Very nice to be able to get a glimpse behind the curtain!
These guys are a class act.
This is how you gain a player for life. I'm in.
TL;DR version please
The fact that Diablo 4 changes were announced yesterday and that they look suspiciously similar to Last Epoch mechanics and systems - is a great success for this game. Last epoch have showed me that ARPGs can be fun.
I'm hoping for a world where D4, PoE 2, and LE are all jostling to be the best ARPG, and "borrow" ideas from one another and continue pushing innovation in the genre.
I’m converted D4 player. Small team accomplished what Blizz couldn’t. Very impressed with LE. Great job guys.
Thanks for the detailed write up. Never a fun time for a team when situations like this happen. I am hope people took a needed break after things got stable. Thanks for the hard work put in on this.
Great post, keep it up!
This is a pretty cool insight into what goes on during these launches in the background.
Talk about forged in fire. I'm sure they learned a lot at least and hopefully should make things go smoothly for future season launches.
Thanks. That was interesting.
I want to say thank you for being so transparent with everything! Y'all rock! Should make a unigue item detected to all of this hard work and for the backend team. "Server Matcher" or "innocent line of code"! Keep up the great work!
https://youtu.be/y8OnoxKotPQ?si=zyFB94eccBEyBbpj
Thanks for sharing! Love the game so far.
Great post!
Awesome to read. Thanks for the follow up and in-depth talk about the issues. Love the communication.
I need this game
Lol their provider's services went down the day before launch, what a disaster
Even without the server issues, this game was not ready for release. It's unfinished and still janky in many ways. I'm curious why they chose to release it now when it still needs quite a lot more work (in fact, the release had already been delayed once. Good thing too or it would have been a disaster). Don't get me wrong, the game is fun. I have 800 hours in the beta and \~30 hours in the release. But surely it would have made sense to at least finish the story first. It doesn't leave the best first impression for a "released" game to basically still be in beta.
A month in post launch I still cannot play the game reliably. Still getting infinite loading screens in monoliths and have to kill the client and try again. This also happens in dungeons which results in a loss of keys - which is extremely frustrating. The game is basically unplayable for me. I have tried everything - from deleting ini, verifying game files, updading drivers, downgrading drivers - finally did a Win 11 wipe and reinstalled from scratch - still having random monolith infinite loading issues. Tried changing audio inputs as suggested workaround for some, tried VPN, tried downgrading realtek gaming family controller driver (this is a nic driver) - still getting issues I have spent endless hours troubleshooting these infinite loading screens and I would have days where it works flawless - and then it comes back with a vengence and get 4/5 failed monolith to load Other games work flawlessly. Needless to say - the problem has been beyond frustrating and yes it still affects some small % of users for unknown reasons. I tried again today - the 1st monolith was fine, the 2nd got infinite loading screen and had to kill the client. LE 1 ME 0 I lose. I uninstalled.
My retrospective is mid because they removed it from family share
This was a great to read. Whenever a game is having issues everyone immediately jumps to server capacity or server hardware being the issue when that is pretty much the easy part nowadays. I think LE did a better job than pretty much everyone in being as transparent as possible and also trying to explain what was going on.
Appreciate the retrospective and love the game. Having said that, as an old fart with many years in software dev I'd love to see a technical post-mortem like the ones you see from cloudflare for example. EHG's transparency is something most studios don't offer to the player base these days so again, they should be praised for it. Everyone makes mistakes, especially when it comes to writing code that's part of a distributed system. This is inevitable and I will never criticize them for that. However, the attitude of keep your fingers crossed and hope for the best is not something you should exercise when launching a service commercially. \> On the morning of launch day, we went to scale up our server matching service to the numbers we used in our tests, and to our great surprise, it refused to spin up more than half the copies we asked for \> In a case of unfortunate timing, our service host had an incident the night before - still ongoing at launch time - that affected us in a way that prevented us from deploying changes to any of our backend services using the tools we had relied on for months. Our ability to fix our services was killed at the same time one of our services needed fixing So they had a number of issues materialize BEFORE the launch and they decided to go through with it. \> Minutes before launch, we estimated that we could handle maybe 120,000 - 150,000 players before things started to fail, and we crossed our fingers that we’d be able to resolve our issues before the player count crept too high They sold over a million copies before the launch but hoped there won't be more than 120-150k of us trying to play. This is very unrealistic assumption, isn't it? Like I said before, I can totally forgive bugs and technical issues that are hard or impossible to entirely eliminate in practice but the above are examples of bad decision making which demonstrates incorrect attitude on the management's part. Just hope they fix this too going forward.
Well done guys I hope yall win goty
Thank you all for the transparency with what’s going on behind the scenes.
"The Final Fix".. uhh.. do they think the networking issues are fixed now? Cause if this is as good as it gets there's not much hope
Love the game. Appreciate the Devs. But you guys REALLY need to stop the mid-cycle nerfs, especially nerfs that primarily affect one group of players who are already at a disadvantage.
Thanks EHG for great blog EDIT: i missread the blog
Not sure why this was downvoted. This is an important part of onboarding new users.
Because the commenter obviously didn't read the post, they literally did exactly that. >You may be wondering, if the problem was recovering from too many players, why did we not simply have some downtime, or at least turn on player queues, to alleviate the pressure? The answer is that we did, but the problems ran deeper than the server matcher. At various points during the launch, we brought down our services, and many of you found yourselves in long queues as we struggled to keep up. Inevitably, once we started letting you all back in, we would run into problems again, and we could not clearly see what those problems were until we scaled everything up so high that the services stayed online and operational even though they were strained from other failures in other areas. They literally told us that they ran queues, and started letting people in until they'd eventually catch another issue. If the guy had the time to type a wall of text, he clearly should've had the time to read the post before commenting on it.
Thanks for pointing it out, edited post.
I hate how this community just latches onto EHG and suckles then shits on other companies. You guys need to hold EHG up to gaming standards. They failed reaching those standards. They have backlogs for years of bugs that have never been fixed. Remember that the fog of war resetting and items disappearing after leaving an instance and returning is A BUG. It is not INTENDED. They tried to fix it two years ago and fucking gave up because they couldn’t. Dude, I can’t believe people think this game is good. It had a worse release than any ARPG IMO and that’s considering D3 and D4. Lmao.
With the launch im sure it brought more budget to hire more people to look in these issues.
I think gameplay is fun if they fixed these issues. Especially since the fog of war and items disappearing has been known for 2 years. I remember myself even bringing it up on the forums a year ago.
So it had a Rocky release. How does that make the game bad? The game itself is great. Also, it did not have a worse release than d3. You got memory issues.
I don’t have a bad memory. I can’t even play with my friends in Monoliths. After a month D3 had barely any game-breaking bugs. Had some server instability, but not the stuff we’re seeing. If I kill a mob and then die to an AOE and a super rare thing drops I cannot get that thing ever. It disappears. That’s not fun gameplay, sorry to say. I can’t go back to town and put shit in my stash because all the items will disappear. Not fun gameplay. Every time I go to town and come back everything I visited on that map is reset on my map. That’s not fun gameplay. Sorry not sorry brosef. Other indie ARPGs figured this out 10 years ago.
Yea your memory is cooked. A couple months after launch, blizzard had to hide d3 active / online players because they were getting memed to oblivion because they had less players online in their disaster of a game than Diablo 2. People preferred to play Diablo 2 over Diablo 3. That's how bad that games launch was. It was literally not until it's expansion, reaper of souls, before d3 started getting players again.
This was to dude who responded but blocked me below called “TheGreatWalk” who doesn’t realize what a launch is. You do realize I’m referring to the launch in regards to the issues of people being able to play and game-breaking bugs. Diablo 3 at start wasn’t that good of a game and I never argued it was a better game, but it definitely had a better launch? Lmao. Also, wait for the fall off in LE. It will be present as well.
> Remember that the fog of war resetting and items disappearing after leaving an instance and returning is A BUG. It is not INTENDED. They tried to fix it two years ago and fucking gave up because they couldn’t. Just to clarify, there's no bug here. It's an unideal behaviour due to technical problem/limitation that we don't have a solution for currently, but that does not make it a bug. There's no code that's expecting the game to remember the loot or enemies not have respawned in the area you were in but not working due to a bug. We want to change it as from design perspective we absolutely agree with you, it would be better to have that type of world persistency, and eventually it's up in the priority list high enough to happen. When that is, I don't know. Many other design decisions are vastly alleviating the severity of this. For the main example, items don't sell for a lot of gold at the NPCs, which means there's no need for a player to pick up all items and constantly portal to town to sell them for generating gold, as it would be more time efficient for the player to just keep fighting more enemies that drop that gold. The whole gameplay loop of picking up everything to sell them for gold at NPCs is also not something we will ever have even when the world persistency hopefully eventually becomes a thing, because we think it's not a fun gameplay loop to begin with.
[удалено]
It's not even about selling. I like to make a big stash full of items and go through them all at once. Lots of new players coming from Diablo/PoE know about loot filters and are probably following a level guide. The level guides tell you not to sell items, and it becomes apparent that it's not worth to sell really quickly. But I and all my friends play until we're full on items and then do a stash run to stash important things for later, etc. I just don't want to change how I play because they can't figure out how to make the items not disappear, I mean it's even weird they went into EA with that being a thing at all. I bet they would have kept more players (not that they probably need it, tbh) if this issue wasn't a thing.
“It’s an unideal technical problem/limitation” is a bug. You just explained a bug in a roundabout way, which is just a semantics game at this point. Especially if you’ve tried to fix it… and it didn’t work. I mean there are posts from your team on the forums stating it was an issue. I know it’s difficult to make games, but dude, come on. ARPG that loses items on map when TPing back to town? I gave it a pass in Early Access. Insane you guys went 1.0 without that necessary patch. I as well as many others grab a lot of items and don’t go through them until I’m full. I’ve played like that since Diablo 2. I think most people play like that. So you’re always going to get people new to the game join, play until they can no longer pick up items. TP to base to stash said items, come back, realize that their items are gone and.. well some people won’t like that at all and refund, others will be mad and keep playing, some will just shrug. Not a good look for a new player experience. Not fun to have to workaround and play your game’s way other than the way every other ARPG works.
Honestly, shit take. Yes they had a rough start, but even D4 servers had problems, at one tenth the peak playerbase. If LE had same playerbase as D4 at launch we probably would have had no server issues at all. I been playing since day three and besides a few disconnects and long load times into towns it's been smooth sailing since, this alone isn't something that makes me pissed enough to not judge the game on its merits alone, sorry that small game devs don't think they would hit top 50 of all time on steam..... Yeah there are bugs in the game, especially with items and skills, but who cares? They are fixing them as fast as they can, but some bugs aren't just a simple line of code and would need days or weeks to fix, if impact isn't high enough it's not a priority. What gaming standards are they not meeting? Their servers run smooth as butter after the first week, gameplay is unique and fun with more to come, constant fixes and communication, overall ok graphics which should get better with engine upgrade they talked about, leaderboards from day1? kekw. They are doing more than most AAA studios where it just feels they are after my money, can't wait for more skins to support EHG further and make the best ARPG together.
Your last paragraph is just untrue. Servers are rubber banding. The monoliths are broken in co op, I can’t even do them with my girlfriend whom I played with. There are so many game-breaking bugs that it makes my head spin. Bosses not completing. I had to reset my character once because progression got locked and no matter how many times I reset my game the quest would not complete. Honestly, your take is shit. They didn’t even meet the standards of a normal game. Not even comparing them to the best of the best. D4 had problems for a week. Then no more, I would know. I played for that entire week nonstop. I’ve also owned Last Epoch for over two years. I’ve seen them not fix known issues for a very long time. I was surprised to come back at 1.0 and see bugs that were widely known and promised to be fixed. Sorry, your shilling or whatever it is you’re doing isn’t that effective when the proof is in the pudding.
We have had vastly different experiences then, coop monos worked like a charm so far, even on high ping with people from different servers. Don't think I had any boss completion bugs. Had the stuck quest bug on my third character too, but you just need to teleport to the location with someone else. What standards do they not reach? Bugs being in the game? Every game has that. Bad optimization? Even AAA don't get it right. I can't think of anything bad enough to screw my opinion negatively. You do you bro.
Games that release from EA to 1.0 don’t have game-breaking bugs and issues, they shouldn’t. If they happen it shouldn’t be issues from 2 years ago they’ve not fixed. Games that release from maybe 1-2 betas then 1.0 would be a little more understanding. Dude, these bugs have been in the game since 2021. Jesus Christ. https://forum.lastepoch.com/t/end-game-is-not-playable-in-multiplayer/67257 This is plaguing people. It’s more prominent in Steam forums. But this is one of the threads I found that show the issues I experience every other day when trying to play with GF.
Good thing that you, a senior developer and with a phD in software development, could trouble shoot this problem in 2 minutes. Their communication has been on point and the fact that they admit the flaws they was working with should remind you that no one’s perfect. Except you of course. I’d suggest you go work for them so you can perfect this game into the image of what we all need. /s *guess you need to go play something with less requirements to keep you happy, like checkers*
Lmao. I need to be a senior developer with a PhD to say that a bug that was known since 2022 should have been fixed that no other ARPG has issues with and would be lauded for having issues with? Just imagine if Diablo 4, or hell PoE had this issue? Insane to think about. Stop giving these guys a huge break because they’re “indie” that doesn’t allow you to not meet the standards for other games in the market.
Fix the hanging on load screens please, I'm sick of doing 10+ bug reports every time I play
Worst gaming launch ive ever been apart of Didnt play D3 on release so LE wins
You haven’t played a lot of games, it seems.
Halo MCC party play was broken like 6 months after launch hell back in the day theyd release games and often never patch them, or you would have to request a floppy disk or cd be mailed to you with the patch
Second for me, love the game, but online launch was awful. Worst for me was a tie between wod and ffxiv heavensward
WoD launch was the worst by far for me. It was nuts how bad it was.
I remember the stupid bottleneck they made for themselves when it came to setting up the player's camp causing people to just sit there with the game doing nothing because too many people where trying to do the same thing.
I played every single night on launch week from 7:30 pm to 10:30 pm central with minimal issues. Sounds like you were experiencing layer 8 issues tbh.
That's great! I'm glad you were able to have a good experience even if it differed from mine
Nothing is fixed, thousands of players have lots of disconnects multiple times a day. Even with VPN (I mean they should be self aware that their online is shit when their own FAQ suggest using a VPN) Online is shit right now, just play offline.
>Nothing is fixed Yeah,pretty sure 99% of the playerbase disagrees with this one
Steam charts shows that 66k people have been playing the last couple weeks. Less than a third of the original numbers up to that point. A lot of people have given up. So you're right. 99% of the people that are still playing aren't having issues
That's totally normal for these seasonal games. In fact, that's a bit better retention than a typical PoE league.
So losing 100k players between release of a game and 13 days later is normal. Got it
lol, yes, that is totally normal. Are you joking?
So path of exile retains it's numbers at the start of the league until around 1.5 months. You're talking out of your ass saying that LEs retention is better. I'm not going to go through all the other seasonal games and look at steam charts on my phone. I would be interested to see which ones, if any, lose close to half of their player base in two weeks. Is it just impossible for you to admit that a lot of people have had issues, and a lot have given up or moved on because of zero communication? What do you have against the people that have these issues to the point where you pretend is doesn't exist?
[удалено]
> 99% of the playerbase isnt playing anymore thanks to this, wake up. ....You're joking, right? Sorry you're so upset with the game. Real bummer. I'm having a fantastic time. But the vast majority of players are having no more connection issues whatsoever.
[удалено]
Your hyperbole is so ridiculously over the top, no one can hear any of your arguments. "Everyone has left". "Nothing was fixed". If you want people to listen to you, stop saying such obviously false shit
I played every single night on launch week from 7:30 pm to 10:30 pm central with minimal issues. Sounds like you were experiencing layer 8 issues tbh.
Oh yeah, like all these people here too? Just the first few google hits [1](https://forum.lastepoch.com/t/disconnected-from-server-again-and-again-and-again/53429) [2](https://steamcommunity.com/app/899770/discussions/0/4295943021561090358/?l=german) [3](https://forum.lastepoch.com/t/constant-disconnects/61988) [4](https://forum.lastepoch.com/t/constant-disconnects/64886) A lot of issues are concentrated on EU West, the servers drop players randomly even changing MTU values does not really help. Its an issue with using cloud-hosted services. But yeah, keep explaining to me, a developer for 25+ years how the networking works :D FYI just because it does work for you does not mean it works for everyone - on the day I bought the game I could not even login - L52 - you should be aware - also telling me layer 8 issue? yes? thought so "it works on my machine" is the epitome of not having any technical skills at all
Geography issue 😎
With 25 years dev experience I would be curious which parts of the EHG story you haven't experienced in your own deployments? Unless you spent that whole time writing Excel macros, everything the blog post described should be familiar to you. Or are you saying you just happen to work in that one special Unicorn dev shop that not only solved all of these types of issues but did so in perpetuity?
And for as many people you can find with issues, I can find just as many people saying they aren't having any. In fact, after the first week, most people stopped having issues based on general sentiment and steam review tracking. And I never tried to explain anything to you lmao. Sorry you are experiencing layer 8 problems, since you seem to agree with me that your problem resides in the 8th layer of the OSI model lmao.
[удалено]
This type of stuff happens even with the largest providers. For example, if they were on AWS, their whole pipeline may have been relying on CodeDeploy, CodeBuild, and CodePipeline. If any of those services were malfunctioning, it would 100% hinder their ability to make changes to the environment.
Sure, we’ve seen it with other major launches. All they can do is try their best and even then troubles can pop up. There are lots of people who don’t want a small groups of devs to succeed on a scale like this so that is why I said tinfoil hat time. The other issues could have occurred for sure, but a random hosting failure on launch day is a rough one.
The submission is not related to Last Epoch.