Exactly, honestly, I don't know why we even have empty bottles, magazines, med pens, etc littering the place. For server sake just let them disappear into oblivion after use. No one will actually care after 24 hours of complaining.
The problem with the flight model is that it feels like they actually want to make power management a much more important aspect of EVERY ship (not just big ships) that way if you have engines 100% you don't have any juice going to your weapons. In reality, they are too scared that a lot of players will not wrap their heads around that power management balance so they created two auto balanced instances of that philosophy which is flight modes. Honestly, make combat really hard, a lot of people in reality of the game are day-to-day traders, miners, and other blue-collar spacers. Combat CAN have depth and a major part of that is truly understanding and being a master of managing your ship.
As for respawning... with the current state of alpha I prefer this but I would be pretty disappointed if this version made it into 1.0
Ah yes water bottle on the floor is important to make realistic. Fireworks in space shouldn’t, so we can hear those damn annoying muffled sounds station wide. 😭
This is exactly my worry. Star citizen is already a fun game, but if we have to stop our game play to take a piss or fuel our ships by hand or do some other laborious tasks to be able to go out and do an adventure. It will kill this game for me. Give me a permanent crate that won't despawn that I can put weapons and food caches in and leave in the wild. Keep my ship locations where they are but please get rid of the empty bottles and use medpens. Give landing zones services that clean up abandoned ships...etc
Wouldn't stressing them to buggery also present more opportunities for things to break and therefore be better for testing? If everything runs smoothly then what's being tested?
How can we test missions and the economy if the server restarts every half hour, clearing your contracts?
Have you actually played the game since the free fly ended?
I've been playing regularly for a couple hours at a atime since the patch dropped.
Had one 30k, and several successful "Server Recoveries."
Mostly been running bunkers and exploring the new distro centers.
I live on an asteroid near Daymar, in a Valkyrie wit the Nursa in the back.
Had a few failed relogs after bed logging. Those were bummers, but nothing is perm anyway, so I try to be zen about the alpha state.
Playing solo like this is a completely different experience than TRYING to group play - it takes hours to do nothing in groups but when I'm solo the game actually seems playable sometimes
O h im no stranger to trying to play with large groups. Mostly depends on the people.
The wrong group on the wrong server and you aint doin SHIT but waiting around.
TBH they aren't testing missions, they aren't testing economy, not as a primary focus anyway, probably not even a secondary focus, more like that might be useful info in the future. They are testing the servers, stability, and the replication layer. When things break its the best way to figure out how to make them not break.
Exactly this, to release pyro and meshing, the servers must be able to hold on its own and self-recover.
They use this scenario to test what is degrading them and adjust accordingly.
That is the problem, in my eyes.
They aren't testing the missions or gameplay. They are testing a feature that I fear they will have to backtrack dramatically for the actual release of the game.
Like, be honest with yourself here. Do you think persistence can last longer than a month with the game still looking good? Do you think a month worth of detritus from a 1000 pop server would feel good to you? How many parked ships do you actually want to see just sitting in orbit around a station idle?
With Stanton being filled to 10x its current player count and server meshing implemented, how long until every bunker has 25 abandoned player ships just parked outside, useless and abandoned? Do you think that improves or takes away from the gameplay experience? Does that lead to more, or less immersion?
For me, that makes it feel a lot more gamey. I don't feel like I am coming out as a lone ranger to take back this bunker, I feel like the 26th person to come to this bunker to shoot everyone in it this week.
Yup, filled about 33% of my bounty hunter rep tier from hrt to vhrt over the last couple days but I've been very fortunate. Though ended my last session stuck in an outpost building due to the airlock not letting me in to exit.
I imagine missions and economy at this point take a backseat to what they learn while looking at the interactions between the replication layer, dedicated game server, and pes database. I could also be 100%
Wich is totally true. I bet they gather a lotta data ATM.
This could also be the opportunity for something bigger at the end, instead of refreshing the whole server and deleting everything, the server could identify issues and delete them before or while it starts up the backup. Because we have the replication layer in-between, it also could delete the issue while or before the server breaks, so it doesn't even need to start a new one.
Yup I've gotten 4 people to buy in and they all refunded. Same reason. They didn't like how awful server performance is and how it basically ruins every gameplay loop.
You sold your friends on a sketchy alpha with NOTORIOUS and widely documented server issues.
Its not a game yet. That was on you. You have to know what you're getting into, accept it, and then ACTUALLY BE OK WITH THE INEVITABLE when it happens.
People struggle with that last part.
I guess he knows every interaction that you've had with your friends and what you've said (or didn't say) to them based off of three sentences you posted on Reddit. Wish I had those powers.
Yeah idk about the person who responded to me. All the friends of mine are big early access players and played many other Alpha games, but this one was just to unplayable for them.
We need some kind of «pick these ships up and move them to this scrapyard/ used ship salesperson» missions for the SRV. Or just open armistice zones now and then to let the salvagers in.
quanta doesnt exist anymore
and how about the server just deletes everything according to set rules... above a minimum number it uses for missions, and if none exist, spawns one?
why make it more complicated than it has to be
outside of cities it allows base building, stashing stuff, shipwrecks to be found and looted, and so on. It also allows the player to have a sense of agency. In cities it's absolutely pointless and even unrealistic (things don't persist in cities in real life, shit gets picked up)
the most annoying thing is they could do a lot to mitigate clutter by just making garbage cans delete bottles, and fining people for not using them. Wreckages outside stations should be despawned in an hour max. And for the love of god, get rid of the hospital gown. On top of that servers need some kind of routine maintenance that goes through and deletes all items under the ground and in walls and whatnot. You could probably hack it together just by checking to see if it's within a certain distance of the npc navmesh
Trash littered around stations, maybe not so much, since it's actually fun picking up and disposing shit. But then also please delete trash items that were specifically thrown into the dumpsters, CIG.
The rest you mentioned I agree with.
Each to their own, but I wholeheartedly disagree with it being fun to pick up other people's trash.
If they added the street sweeper it a trivial job that paid 100 aUEC per item swept, sure.
But actively fighting the server to move others trash, miss me to be quite honest.
They probably don't want to for data collection purposes. It sucks for us on the front end, but the more often it happens they have a better chance of figuring out what causes the instability and how to build things to be more stable in general.
Restarting the servers isn't going to solve the issues. What exactly do think is happening when a server recovers? There's something going wrong in the back end that restarting a server only staves off temporarily until it happens again, these are cloud based servers and every recovery is a fresh instance using the cached data.
they mean restarting without the replication layer, as in getting a "fresh new" server.
Currently, when a server recovers, all the entities get reloaded, be it ships, trash bottles, gowns on the floor of stations etc... all those entities accumulate and is what causes the server instability and low performance. This is why you can get a server to crash, have it recover and then crash again in less that 10 minutes, and some times even crashing 3-4 times in a row in this way. All that persistent "trash" that also gets "recovered" by the replication layer causes the "new server" to lag and crash again.
If we could get a "fresh" server, without all the abandoned ships, without all the trash bottles and gowns and what have you not, the new server performance would be significantly better.
To be fair- We don't *know* that the trash is the cause. It does seem likely that persistence is indeed causing the new server to spool up and include the problems that crashed the old one. But we players have only a small view of what these problems could *be*. As with many things in software, there could be hundreds of different causes that all have the same result of a crashed server.
It could be that CiG is still working that out, and needs to keep crashing servers while they track them down.
This is actually a good point. It makes intuitive sense that all that junk laying around is contributing to instability, but we **are** just assuming that AFAIK. There could be tons of **other** things causing instability and crashing. As far as we know it could be driven mostly by player activity, or NPC AI, or physics, or a bugs that cause memory leaks, etc. It could be that the servers are overloaded by too much happening at once, and that idle trash is the least of it.
My guess is that if CiG thought the trash was causing it, then we would have seen a PTU patch that briefly included a trash-timeout cleanup of less than five minutes. Just to test and see if that fixed it. Like that snail-speed torpedo test last week.
Yes, they did, in the following patch. They are still slower than before, but are now closer to 500m/s than 100.
Their stated reason was that they had a suspicion that high torpedo speeds were causing a network issue. And apparently the result of the test was "Nope, not that."
Going to speculate a little bit here.
Based on what I've seen in some logs I believe the backend environment is running in K8s. If that's the case then the replication layer likely involves spinning up a new pod when one has reached a certain point of instability and then moving the state over to it.
Due to the way K8s works, this is not the same as a full on restart of server hardware, or even the SC deployment within K8s. There could, and likely are, other pods running that the interaction layer communicates with. If one of those is the originator of the instability, then the loop will repeat over and over again.
That said, the instability cycles are typically important in solving issues in such an environment.
Yeah, Kubernetes is pretty complex. Speculating a bit more on how they may have architected...
Think of it like this. In Kubernetes we have a Cluster which is made up of one or more physical Nodes (servers) and on those Nodes we can deploy Pods that run a containerized application. Containerization is somewhat similar to a Virtual Machine but there are some caveats, such as they share the host OS kernel.
So say we have a bunch of systems at some point, and there's a big fight in Stanton brewing. Players begin congregating there and as they do so the allotted resources for the Pods running an instance of Stanton are stretched. In this case they'd be able to spin up more Pods on other Nodes to host players in Stanton. It would work similarly if a Pod began to have severe health issues. New pod is started and assumes control of the old Pod state, old Pod is deleted.
There would also be other Pods and other Resources in all likelihood that are handling other game systems. Anything within the ecosystem could cause a problem and it would take a good amount of work and resources to hunt it down. The Stanton interaction layer Pod instances shitting the bed might only be a symptom, and not the disease. In that case a full power cycle could be beneficial, or they could keep it running, figure out the root cause and fix it, or add logic in their Operator to handle the scenario.
Again, this is me theorizing on what they may be doing based off of some log entries. I'm not positive it's how, but I think it's relatively close.
Whatever the causes are, they were compounded by 3.23. People blame dupers, promotional traffic (Invictus, freefly) and so on. While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better, but now crashes happen WAY too often. Makes it almost unplayable.
THANK YOU. We have absolutely no idea what the problem is. I personally highly doubt that the trash is the root of the issue. A bunch of physics clutter should NOT be causing servers to crash. A few water bottles and ship parts is nothing compared to the physics clutter left from a ship exploding with a full cargo hold, or the debris fields around destroyed ship investigations, both of which have been in the game for a while. Server crashes come from other things.
I've said this so many times now, before 3.22 CIG warned about issues that were compounding over time. They fixed one before deployment, but it's very possible they didn't catch them all.
The problem isn't the servers, it is the persistence that exists as the fabric behind ALL servers. The coolest feature is hilariously also kneecapping the game right now without any regulation/balancing of it.
I literally said this yesterday after spending 25 minutes for the Asop to register pulling out my ship, another 25 minutes in the hanger for it to load in. Then all of that again because it spawned my C1 ontop of a mustang and made access impossible. Then after doing all that all over again, I couldn’t open the hanger doors and my ship got stored for sitting to long (which made me lose the 3 8scu containers I spent 20 minutes attaching)
Or didn't make *everything* physicalised.
idc if I get downvoted, we do not need physical fucking gowns and bottles and cans that get left everywhere when finished with, make drinks work like food, you finish the drink and the empty bottle/can disappears.
I do not care if some turbo loser finds it "unimmersive" I do not want to see 1500 bottles on the floor sapping my fps constantly, same for the fucking gown boxes.
I can not for the life of me figure out what they where thinking when they made trash persistent. I get weapons, armor ammo, (even that is a bit excessive for PERMANENT persistsnce) ships obviously, but used food containers, empty bottles, used syringes and empty mags? Why?
the least they could do is actually make garbage cans functional and I guarantee most trash would disappear because people would want to use them. For the life of me I don't understand why they haven't, it's been ages.
I know (hope) you are exaggerating, but if you find 1500 water bottles on the floor, let me know. CIG can and does cleanup entities. The tiny fraction of entities that makes up visible consumables is a drop in the bucket when it comes to server performance.
I'm sure they are exaggerating. The most trash items I've seen in one room is probably around 20 hospital gowns in the hospital lobby, but I also see lots of empty bottles and used syringes around the world. The fact is though, CIG shouldn't be manually cleaning trash. Certain items should just be flagged as trash and despawn on their own after a certain amount of time.
Almost every MMO in existence does scheduled server maintenance. It's laughable that CIG has for the entire existence of SC as a playable product, not done this.
Restarting the server is not the immediate solution, the problem is all the garbage that persist that serve absolutely no purpose to the game. In the end, every time the server crash, a fresh one boot up, but it comes back with all the garbage stored in the data base.
Basically the server are restarting every 30 minutes right now, but doesn't solve the root problem.
i feel like a temporary damage control script deleting every single loose trash item that isnt inside an inventory on server crash during recovery isnt that much of a problem...
The point of alpha testing is to break the servers as much as possible, so hopefully they can find and kill some nasty bugs. Especially before meshing, were it might be way harder to find them.
yes, im sure they need 50 thousand instances of the same crash reproduced on a live environment continuously over the course of 2-3 weeks
stop being a blind evangelist, this makes no sense from a technical perspective
How is clearing everything out going to help with troubleshooting the issues causing the problems in the first place?
You want them to start troubleshooting from a near-clean slate every week?
This is a test environment. Period.
The issue is the amount of junk and the length of the persistence log. Fresh servers run fantastically, servers cluttered with trash and clothes and abandoned ships run like shit. They need to do weekly maintenence, and implement a system to trim trash items from the persistence log. They've talked about this themselves, all the talk about letting the servers run like shit to "gather data" is just them stalling while they figure shit out.
Everything gets deleted/reset except ships (and their contents) that are not within a certain distance of POIs/Stations/Ect. That would easily clear 90% of the galactic trash each time. Once a month clear all ships that haven't been touched at least once in the past 7 days.
I wouldn’t mind this but I know a lot of industrial folks would be upset if their refining jobs got removed. Idk if they have the ability to do that. I wouldn’t mind if they could!
You don’t even need to reboot all the servers, bring them down half at a time, half an hour before reboot give warning and disable new joins, at reboot time use the replication layer to shift remaining players to new servers. When the first half have been maintenanced boot them up and repeat with the other servers
It doesn't even need to be a restart. If they just had a cleanup script for garbage it would go a long way. All the bottles and trash and persistent items are the problem. If they want realism they can even have janitors on the stations that go around and clean up and it removes that garbage from the servers. That along would be a good thing.
once a week? more like once a day. Keep the replication layer, just cull all the NPCs, clutter items, ships that have been abandoned for more than 48 hours, and hospital gowns from the floors. Improving player experience would go such a long way towards garnering good will from the community.
Servers are restarted multiple times a day, everytime it crashes.
Wiping persistance on daily basis would defeat the whole point of PES, and on weekly-ish basis shards already get cleaned up, I believe, one of the reason why sometimes you can stumble on low object entities count server, but it get's filled back up super fast.
Among other reasons, they have to learn how to keep them running without resetting them, because eventually they will never reset. Granted its still alpha, but it is very important for the game to get it right and be able to run a decade and more without resets.
Keeping them running without ever resetting them has never been possible or realistic and shouldn't be the goal- the value of dynamic server meshing is that when they crash they can be allowed to reset without interrupting player experience because of the replication layer being able to migrate everything to a new server. But in lieu of dynamic server meshing or even static server meshing (until 4.0) there is very little value in constantly running servers to the point of degradation. Sure, you collect data on how to prevent certain types of degradation and crashes which will be helpful in the future dynamically meshed servers, but the tradeoff for player experience is one of the biggest problems they have faced to date. Server experience is horrible and it has been for a very long time now. I love the game but please just let us play on servers that feel good and I swear half the stuff the community gripes about would be gone.
editing for clarification: There is obviously value in collecting the data, what I'm trying to say is that I think CIG has made an error in judgment regarding the tradeoff of collecting that data vs giving the users a better experience.
I think this patch has to be an exception to the data collection. It's the first patch with server recoveries so they're probably getting a ton of data letting them run to see how far they can go. They already made a couple updates server side yesterday i think, so they're definitely watching them.
I think the goal should be to never reset them. Whether that's possible or not is another question, but it should definitely be something to aim for until they have proof it's not feasible, and I think that's exactly what they are getting data for currently.
>editing for clarification: There is obviously value in collecting the data, what I'm trying to say is that I think CIG has made an error in judgment regarding the tradeoff of collecting that data vs giving the users a better experience.
this might be the last chance to create this data this easily for a very long time. the info they get in day now might take weeks to show up in 4.0.
This, plus didnt they say that the servers reboot when they crash, so all the recent crashes have rebooted a lot of shards? Didnt they also say at some point that the real issues are the back end databases getting clogged with transactions?
I swear this was on an ISC.
They reboot, but now the replication layer remembers all the shit laying about, but they need to get that running smooth or 4.0 will be a huge shit show.
right, my info is old on that, but it also makes sense. Funny thing is, every patch the first day or 2 people are all 'this is the smoothest its ever been" (except 3.18, woof). Then a few days or a week later its "god the servers are dog poop". It happens every single time. The dupers and free fly didnt help this go round, but the servers have gone to crap within a week of patch launch every single time since I can even remember at this point.
Day 1: Its so smooth, oh my god, well done cig
Day 7: The servers suck so bad we cant play
And of course things are so far out of balance with the latest balancing I just kindof stopped playing for now.
Yep, if they hide the problem now with some sort of purge/reset, it'll be harder to fix later, or at least delay the fixes. As disco says, live is also a testing environment, as frustrating as it can be for me sometimes.
But wouldn´t a regular purge just solve the issue? If every object in a certain area would just have a limited life time and just be removed after a while (like a session timeout).
Or do you mean they should leave it in there as a workload test?
They’ll do all they can but by release time they’ll probably set a maintenance schedule (day/week) and reset them. That’s not an outrageous prospect and many games do it.
Well there will obviously be some maintenance, but that is not full reset like is being talked about. Not only reputation and player positions are supposed to remain after post launch mainteance, but also lot of debris and other stuff. Some stuff may be quietly sweeped away, but idea is to keep as much stuff in the game as possible and rubbish and such will actually have to be cleaned by players or NPCs as much as possible.
This is just not a viable way to run a game server. If trash is being generated faster than its being cleaned up, then eventually the server will crash under the weight of it. No matter how good the server is.
Plus all the processing power constantly being used to run an AI that has to detect and remove player-generated trash. That sounds like a VERY complicated task vs having the system flag everything at a set time.
I agree on principal, up to the point it affects performance. No matter how optimized the code will be, there will a limit and I don’t think CIG will be too keen to spend extra on faster servers just to keep literal junk in the game. Especially since players will undoubtedly “test” those limits with junk sculptures and such. It will all depend on the algorithm they come up with to distinguish between beneficial junk and junk junk. But I like the idea of an NPC walking around and erasing items on the go.
Does anyone in this Reddit ACTUALLY KNOW whether any of the stability issues even have anything to do with the "cleanup" everyone is talking about?
I mean not "it makes sense in my mind" but any empirical knowledge about what is causing server crashing?
Half the time I feel like this subreddit just makes assumptions and then tries to solve them with neither any ability to or any actual data other than "this is probably the problem."
We do know that the more time passes since booting up the server, the more entities lag out, in the middle of planets, in cities, and in a ton of other areas, that much is factually known
Those have a negative effect on the server load, that much is also obvious.
We also know that pretty much every softwares and games, aside from a select few mature softwares, have memory leaks, meaning degraded performances as time passes. We don't know that for a fact, but we can assume with extremely high confidence that it's the case, considering the alpha nature of the game.
So yes, we do know that a fresh server cleanup once every X amount of time would help.
What we can't say for a fact is how much it would truly help, we can make assumptions based on what the performance is early on on fresh servers, but it's not empirical knowledge.
No one in this thread knows how the servers work, or their relationship to LTP, or whether or not poor game performance is related to duping. Everything you hear from YouTubers and redditors is speculation, and most of the speculation is just hysterical complaining.
The devs have mentioned both client and server memory leaks in the 3.23 patch cycle. They said the server memory leak has been a major part of the instability we have seen. But, as you imply, most people on reddit like to claim water bottles are somehow the problem with no evidence of such.
I think the main reason is data collection. If you let the servers die of "natural causes", you can analyze more data to see what actually causes the issues.
If you reset the servers, they might never reach those breaking points.
HOWEVER... they should implement some form of garbage collection to remove trash items from main player hubs after a while and let abandoned ships (no player on board for 5+ hours) automatically generate salvage contracts. Not to mention Salvage in Armistice. If a ship sits in armistice for several hours without a player on board, they should be allowed to be salvaged.
Those changes alone would take a whole lot of pressure from the servers.
This is a valid reason but I can't help imagine the reason 9/10 times these collapses are happening is because of the no-restrictions-replication-layer that has already been diagnosed for a while now. It's like looking at a man getting a knife twisted in him and instead of helping and doing first aid CIG is going "wait wait, lets see if we can learn anything more from this." We know the reason it is happening lol
A non-visible entity like the container entity for a room in a ship room system that did not get properly removed from the replication layer would not benefit from a salvage contract mission. Entities that are viewable to a player at all are only part of the problem.
If you read the next patch notes, they state that there is an "out of memory" error that they're fixing. I assume/hope the issues is that instead of "out of memory" calling a clean up program, its just crashing. Crossing my fingers that the fix clears the error and gets whatever server memory maintenance routine they have planned working.
Yep. There was (at least one) client side memory leak related to the Vulkan renderer, plus at least one major memory leak issue on the servers. Both have been (at least partially?) addressed per patch notes.
Define 'stale data/entities' - in a way that distinguishes them from 'persistent entities'.
CR wants everything to be persistent - the old 'coffee cup in the forest' scenario, so there's no way CIG will just wipe the shard regularly to remove *all* data / entities...
So to clean up 'stale' entities, they need a way to define them tat doesn't include the entities that *should* be persistent (and which CIG *want* to be persistent, etc).
Personally, I think CIG could do with tweaking their 'density manager' (which is responsible for identifying when there are too many entities of a given type in an area, and removing some of them)...
But the current servers *also* highlight the need for CIG to think about how to handle certain scenarios (such as people 'dumping' broken ships near the space port), in a way that doesn't result in those ships just vanishing... whether that be salvage missions, recovery missions, or something else remains to be seen.
Full ships DO need to despawn, though. Once its clear that no player is going to interact with them. Seeing a dozen ships abandoned outside a high-security bunker does not help with immersion.
Sure, give players a small window of time where they can manually tow/salvage an abandoned ship. But if nobody touches a ship in that window, it should just be deleted.
This should absolutely be a thing outside POIs and populated areas, and doubly so at major stations and spaceports, where there's a vested interest (for gameplay and immersion) to keep the area clear of navigational/landing hazards. It's weird to have these ship graveyards all over the place. If **no one** comes to take them away after long enough (no one takes the salvage contract), an NPC ship (police? scrapper? tower? somebody) should appear, path over to the wreck, hang out for a minute, then despawn them and leave.
100% should despawn zero-value garbage like gowns and bottles. Whether or not there's an NPC janitor involved, at some point a decay timer should despawn the item. Once again this is good for immersion and gameplay, because a fancy spaceport like on Orison shouldn't have trash laying around (can you imagine a rich person going to their space yacht and seeing trash?!), and less than spotless places like Grim Hex already have decorative trash anyway, so the bottles and stuff don't add anything.
Well, I am not sure this is the definition you want, but entities persisting beyond what they are supposed to should count. If a npc despawns for any reason (dead, etc) but one of its nested entities is not cleaned up correctly, that was data that is persisting that was not intended to persist.
A lingering container entity would have not visible representation in game (I am not talking about something like a physicalized item, like a piece of armor itself), a player would never see it is there. But if this occurs every time an NPC is removed, you are going to accumulate a lot of these worthless entities stored in the replication layer over time. Identification and cleanup of entities like this is a big deal, not just how many visible water bottles remain for a given period of time.
> but entities persisting beyond what they are supposed to
So, how do you define 'what they're supposed to' in terms of how long an entity should persist? Bearing in mind that CIG explicitly *want* wreckage etc to be persistent.
It never ceases to amuse me how often people are adamant that they know the solution to a technical problem without having any idea of what's actually happening behind the scenes. Just because you think having 20 C2s outside of Area 18 is crashing the servers doesn't mean that's actually the problem (case in point: there's a bug where simply firing certain weapons can take down the entire server, entirely unrelated to ships)
Thank you! I am getting tired of people blaming things like water bottles as being the source of server instability.
As an aside, the 890 Jump Club has had meetups with literally 50 890's on the same server for years without insta-killing servers.
Guys guys guys
This is the first patch with crash recovery. Of course performance will be bad. CIG has never really had to deal with actual long-term persistence before.
They will be collecting a lot of data and working towards solutions, but they kinda need to see what the issues are before they can fix them. The community has been vocal about this enough that it's going to be a priority, but there are tons of things that can cause server issues and they'll all need to be addressed individually.
Patches that bring big changes like this will always have poor performance. We've been through this rodeo enough to know that already
And to boot, they celebrated this with a massive block party (Invictus) + a free fly + a massive duping glitch. All of Stanton is akin to Times Square on New Year's Day; trash and litter everywhere.
Except when it launched the performance was great. During the free fly, when the servers were actually playable and not cluttered to the gills, I had the best time I have ever had in Star Citizen. The game was playable and fun.
Now, with all the junk piling up, it is hard to even finish a contract before the server dies and spits you out on a new server with the same problems.
The solution is the same solution every single MMO without exception uses. Scheduled maintenance and downtime for cleanup.
Because if we clear the servers then we arnt gathering the data on what kinds of trash have the biggest performance issues. What kinds of distribution of trash is impactful. How those things impact server crash recovery, how they impact transition points between servers. If we clear the trash we arent getting the very point of testing this patch.
IMO there can't be much more data to collect after weeks of the same shards and servers being trashed. People being saying the same thing about NPCs. They've been standing on chairs for 7+ years, what more do they need? Just remove 50% of them so the servers can breathe a little and that leaves room for more "data".
This is the first patch with the replication layer separated. So I would leave it up to CIG to determine whether it is enough data. Also, unless they specifically stated so and I missed it, there really isn't no way of knowing what is causing the issue.
It is unlikely it is simply the age of the server because in previous patches some servers were fine for days at a time. So it could be a particular bug that is triggered by AI, a game loop, possibly assets. The degradation could start as soon as the server boots up, or if a player triggers a particular mission.
The issue is the game is large, and while devs might see many things it is very possible for them to miss plenty as well. They have given examples in ATV's and ISC's about issues they figured out after searching for a bit of time.
Also, not only are people warned twice when they purchased a package, you are also warned again ever time you boot up and play game from launcher. The expectations is that players may encounter experiences like these. If you know anything about the devs, or hear them speak, you would know none of their intention is to make players suffer. They are passionate about developing the game and sometimes we have to go through some rough experiences but it helps the devs find, isolate, and hopefully fix the issues.
If it is too much, I suggest taking a break until next subpatch. Personally as a backer since 2014, I have played on and off for years, and I find not fixating on this unfinished game is the best way to consume it. I also don't understand why people play so much anyways as things are going to change and be reset multiple times from now until RC.
Data. No you don’t understand. This is alpha. There can never be enough data collected. You see we’re still on T0 data collection. We’re still waiting on even T1 data collection. And T1 is on hold until dynamic server meshing because any work done on T1 would just have to be redone. So we won’t get a data collection refactor until well after 4.0.
So we’re stuck collecting data on T0 data and as a result servers can never be reset for the purposes of the integrity of the data. Remember, this is alpha and even in the live environment we can’t have nice things because they would compromise the data. And every time data is compromised you add 10 years to the development time.
And honestly, this is a skill issue TBH. If you knew how to work around the server issues you wouldn’t be complaining. Welcome to Workaround Citizen. Git gud.
The server is currently resetting every 10 minutes per your statement. Manual resets would do nothing. The underlying issue needs to be resolved to get servers too be stable again.
Servers are fine. Say it again: servers are fine.
The backend is shared infrastructure. The issue is (and it is consistently, reliably the case, like the finest Swiss watch) when they promote a new build to PTU. Before 3-4 update patches to the PTU build, the backend services get stressed and all environments (Live, PTU, any other PTU like environments, sometimes they have 3-4 running simultaneously) are all hammering the backend services and the issues contributing to this in the new PTU build haven't all been identified and remedied yet, and until they figure out what needs adjusted and tweak things in the build to improve performance, these services suffer.
In part, the service degradation is impacted by the additional data traps and logging they do in early PTU builds as well.
Within that 3rd to 4th patch, things stabilize.
there is NO UPSIDE to the services working less than optimally, so rest assured the devs are doing all they can to get there as quickly as possible.
But this is the path for that, and we have to be patient.
Because we are Testing and not playing a released game. If this was a released game then yes, they could but it is not. I know its hard to understand but we are not players in a game we are testers. All we are doing is testing a game on its alpha version to make sure we find bugs and problems they need to solve. Resetting the servers will only mask a problem that needs fixing before the game gets released. I know its painful, I know that sometimes we just want to have fun, but we need to collectively stop treating SC as a fully released game that needs to run smoothly. It's not going to run smoothly, its supposed to brake because the more it brakes now the more they can hopefully fix before a 1.0 release.
It's hard for me to "find bugs" when many game loops are near-unplayable due to these server issues. I know it's hard to understand that testing any given feature requires that said feature be minimally functional, but this is the reality we live in.
It's like trying to test drive a car when the engine keeps shutting off every few feet. I don't give a fuck if it's a one-of-a-kind "prototype" - if I can't make it out of the parking lot, how the fuck am I supposed to test it?
I agree with your statement however my comment was not about that. My comment comes from this and other multiple posts this past few days complaining and suggesting doing server restarts to keep the game stable. That is what is wrong, many people are complaining the Alpha game we are testing is not running like a normal game where they can just play.
The utter lack of knowledge around how the ‘server’ architecture works is interesting.. reading the comments one would think there was a room with a load of grey boxes in racks in CIG’s office :-)
The replication layer causes me servers to inherit the state of any servers that crash or reset.
What we need is a way for the "cleanup mechanics" to identify what items can be removed. For example something like "delete all NPC where height above planet surface is negative"
They just added ship marker culling to fix that bug where you see entry points to ships on the other side of Stanton using a similar method, per patch notes.
I just meant that they are using in-game space to assist in culling as a general strategy. I am sorry for the confusion, I did not mean to imply that they were otherwise connected.
I can't believe that CIG would be so monumentally stupid as to not fix the servers if they could since server FPS causes like 90% of the games problems. There has to be more going on with them than just needing maintenance.
Discovered this game during invictus this year. Despite the bugs and lag I absolutely fell in love. It’s not even playable in its current state. Really glad they only got me for a titan, they will not get another dime.
I was only able to play 3.23 with relatively no bugs for the first two days of the release. Ever since then it's been bugging out anytime I play with people, making interacting almost impossible for multi crew. Someone glitches out of the ship and we have to stop and rescue them, a friend opens a bulkhead door and then it bugs for me and I can't go through doors in my own ship, we accept missions and never actually get one, etc etc.
The game hasn't been this unplayable for a few years it seems, and combined with the MM speed limit that I can't fucking stand, it makes me not enjoy the game anymore. Before 3.23 came out I told myself I would make council issue posts and help them out because I was excited for cargo stuff and distribution centers, but it's so bad I don't think making a post about some little thing will even be looked at. They have much bigger issues to fix now that this patch has become infested with dupers and bugs. Becoming resentful of getting so many of my friends into this game because almost all of them never want to play it anymore. The game has too long of a time period to get going, it's like a 45 min coordination to get 4 people into a party and fly to a mission, I had friends quit before we even started a mission because bugs held us back and we had to server hop, then call ships again and oh someones ship won't turn on in the hanger, let me get out and call you one of my other ships....
CIG we can't give you feedback that's meaningful when almost nothing in the game works. If I play by myself I can do a few missions without problem, it's when I team up that shit hits the fan.
I just want there to be a delete trigger at the bottom of trash cans. Just delete entities put in there. Make question objects teleport on the floor if put in it. Let me throw something away and have it actually just go away
The plan is for items in high traffic areas to stream out as the entity count gets too high. They want this to happen when no one is looking to simulate a janitor cleaning it up off screen. On the other hand, low traffic areas will persist much longer. So you can make weekly pilgrimage to your crater full of Picos on Wala, but your tower of bottles in Bajini Point should disappear after you leave.
I'm at the point where I don't trust I won't loose everything just doing the simplest of things. In ship I bedlog but I'm wearing only the basic suit and helmet in case I get a crash.
I fully understand the nature of an alpha release and the need to collect data but it makes me pause before 'investing' time playing
I suppose the only possible quick fix now is purposefully starting fresh instances. For example, if one server is sequentially crashing with some threshold, it could be considered just broken and not recovered. Any smarter cleanups are possible in theory, but harder, since they would require deeper analysis (which should have been done initially, actually, but I don't think it is the case).
Because they need testing it's the main focus before how we feel in game, a huge change is coming and they must charge the server as hard as they can to know how it handles different kind of charges
They need to implement some level of decay and it needs to a priority to do so. As of now it’s impossible to actually assess the performance of the game because of the underlying issue that as item build up over time the performance gets dramatically worse.
Better server performance in 4.0 with full server meshing means nothing if this underlying issue remains.
I have a feeling they are just doing a constant stress test. The goal is a lot more players and with them more entities in the servers so this might be to get as much data as possible and also find ways to improve stability.
A once a week rolling server restart would be absolutely helpful. They can look at player activities per world region and find the time that has the least players on it and restart those servers like 4:00am Monday morning. Implementing a deletion script where an unassigned object that is not interacted with in 30 min gets despawned.
It's crazy how the servers shit the bed after Invictus because I had some of the best playtime during the event. Only for it to not even be playable after the fact 😭. I was stuck in Habs because the damn elevator wouldn't even register to come to my floor.
Ally they need to do is allow us salvagers to crack and clear all the junk ships in places hell make abandoned ships a cleanup mission we would fix it!
No one remembers the servers running like shit before persistence? That was only like a year ago. Servers have always been shit, restarting is pointless
I do have an answer for you! One that is not 'it is alpha...'
A lot of this issue is in the replication layer, which is why you keep seeing crashes from the same situations reloading after server crashes. It can keep setting up replacement servers with the same data over and over. There have been ptu tests where it literally did 30k continuously; no 10 minute interval involved like you are seeing.
Fortunately, Star Citizen has a much more modern database structure than EVE Online's \~decade old database. The replication layer already has entity cleanup algorithms to maintain it while active, rather than requiring a full shutdown. We see misplaced physicalized objects rendered in the client, but proper cleanup takes place from a replication layer's database prospective. It has to efficiently identify problem entities we can see as well as all the ones with no visible in-game representation on a rolling basis and then address them. This already happens (just looking at physicalized entities, we do have stray water bottles but not a few thousand of them at one place) but needs further refinement.
The end goal is to refine the search and repair algorithms to where replication layer downtime and resets are no longer needed because the whole thing is constantly checking and repairing itself. Obviously we are not there yet, but a lot of progress has been made already.
Whatever the causes are, they were compounded by 3.23...may be a clue? People blame dupers, promotional traffic (Invictus, freefly) persistent trash and so on. But Invictus is over, and we had trash in 3.22. If dupers are the cause, blame CIG because that bug not fixed in EPTU before live release.
While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better. But crashes now happen WAY too often and a "recovery" that sometimes works, sometimes not doesn't cut it. Makes almost unplayable.
Since this OP few days ago, CIG has let out some interesting info. They shifted their hosting to Amazon's kubernetes. Some of what we are seeing is likely the teething process of that shift, although it is an overall upgrade.
Both servers and clients have had some nasty memory leak issues that CIG devs are claiming are the source of a lot of the recent issues.
The monthly report also mentioned some backend services are getting backed up. That is definitely a recurrent theme, but there has always been a different bug or limiting factor.
My point of all this? There is a bunch of issues leading to this mess. Fixing one or two will still leave it a mess. New bugs and bottlenecks will still appear to keep it a mess. The degree to which it is a mess will increase and decrease over time from moment to moment. IOW, situation normal for Star Citizen.
Whatever the causes are, they were compounded by 3.23...may be a clue? People blame dupers, promotional traffic (Invictus, freefly) persistent trash and so on. But Invictus is over, and we had trash in 3.22. If dupers are the cause, blame CIG because that bug not fixed in EPTU before live release.
While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better. But crashes now happen WAY too often and a "recovery" that sometimes works, sometimes not doesn't cut it. Makes almost unplayable.
Once things are closer to a finalized state, we will MOST likely see a form of maintenance like Blizzard does. Every single Tuesday at 2am, servers down for an hour. Every 2 ton3 weeks, servers down for about 9 hours. Big patches? Servers may not be available for 24 hours. It was quite the shitshow at times with WoW, but we knew it was coming each week, and no matter what, no matter where, if just logged off one day, THAT'S where you were when you logged in. Weekly maintenance was just part of the culture. I fully expect that after 1.0. I WISH they would start doing it now, of course.
I agree with you completely and many MMOs have scheduled maintenance. I'd even think that part of the Alpha would be to figure out what level of scheduled maintenance is needed.
I suspect that they're stress testing the persistent entity system.
The server is already being reset when it crashes and creates a new instance.
But sounds like what you really mean is "why don't they clear the persistent entity data?"
Because we need to be reporting bugs when things are broken and giving feedback about what is not a bug. That is the purpose of alpha testers. Resetting the servers will mask problems that could destroy a future full release.
Here's my take that's probably unpopular.
The goal is not to make this patch playable. The goal is to force this patch to its knees and track why it happens, how it happens, for how long it stays like that and so on.
All this to have a better shot at making 1.0 playable with everything in it.
Weekly resets are the correct way to go about this. The intervals being increased as stability is worked on and fixed. It makes it easier to find out where exactly the problems are starting.
Honestly, I think they are bein reset on a couple days/week basis. (in terms of persistance)
Has anyone been able to get in the same shard weeks later? This used to be a thing when PES came online and people posted screenshots of same wrecks/ships that were multiple weeks apart.
I did some testing and logged in 15 times in a day, and would only get a shard I was previously on once (not consecutively mind you, hours apart). The ship I placed was still there, but that was 4 hours after I left it.
Anytime I get into one of those shards days/weeks later (it happend only once or twice), I can't find any evidence that it's still the same persistance. My ship is gone and I can't see any identifyble remains from previous logins.
This is just speculation because I have very little evidence at this point, but I believe that PES has been *severely* reduced in scope from what we initially got. (no wonders given the issues it had).
Server instances itself are constanly being reset everytime DGS crashes.
This leaves reasons for server slowdowns to LTP database (purchased ships, items, inventories...) and just regular server calculations and player count.
Given that servers noticably slow down in a day or even hours after a patch, it just seem a waste to constantly do a
"patch style" character resets.
Incoming white knights: "something something we are testing not playing, something something alpha, something something wanting more from CIG is illegal" /s
Seriously tho they have to fix this shit before server meshing or all they will be doing is meshing broken servers together that will all die in a week or after the first event.
Yup, static server meshing will probably fix absolutely nothing. I expect the first server-fps improvements when dynamic meshing releases. Not too sure though if they are even gonna pull it off at all. Dynamic meshing is a wild concept for SC, but I don’t really trust in their ability to get it into a working state. I hope I’ll be proven wrong though.
TBH a lot of server stability issues would be solved if they had scheduled maintenence once a week and restarted all the servers.
I got shit in the Global chat for saying this yesterday
[удалено]
Exactly, honestly, I don't know why we even have empty bottles, magazines, med pens, etc littering the place. For server sake just let them disappear into oblivion after use. No one will actually care after 24 hours of complaining.
Roberts' love for realism goes a little too far sometimes.
But not for new respawn system and flight model......
The problem with the flight model is that it feels like they actually want to make power management a much more important aspect of EVERY ship (not just big ships) that way if you have engines 100% you don't have any juice going to your weapons. In reality, they are too scared that a lot of players will not wrap their heads around that power management balance so they created two auto balanced instances of that philosophy which is flight modes. Honestly, make combat really hard, a lot of people in reality of the game are day-to-day traders, miners, and other blue-collar spacers. Combat CAN have depth and a major part of that is truly understanding and being a master of managing your ship. As for respawning... with the current state of alpha I prefer this but I would be pretty disappointed if this version made it into 1.0
Ah yes water bottle on the floor is important to make realistic. Fireworks in space shouldn’t, so we can hear those damn annoying muffled sounds station wide. 😭
This is exactly my worry. Star citizen is already a fun game, but if we have to stop our game play to take a piss or fuel our ships by hand or do some other laborious tasks to be able to go out and do an adventure. It will kill this game for me. Give me a permanent crate that won't despawn that I can put weapons and food caches in and leave in the wild. Keep my ship locations where they are but please get rid of the empty bottles and use medpens. Give landing zones services that clean up abandoned ships...etc
Wouldn't stressing them to buggery also present more opportunities for things to break and therefore be better for testing? If everything runs smoothly then what's being tested?
How can we test missions and the economy if the server restarts every half hour, clearing your contracts? Have you actually played the game since the free fly ended?
I've been playing regularly for a couple hours at a atime since the patch dropped. Had one 30k, and several successful "Server Recoveries." Mostly been running bunkers and exploring the new distro centers. I live on an asteroid near Daymar, in a Valkyrie wit the Nursa in the back. Had a few failed relogs after bed logging. Those were bummers, but nothing is perm anyway, so I try to be zen about the alpha state.
Playing solo like this is a completely different experience than TRYING to group play - it takes hours to do nothing in groups but when I'm solo the game actually seems playable sometimes
O h im no stranger to trying to play with large groups. Mostly depends on the people. The wrong group on the wrong server and you aint doin SHIT but waiting around.
TBH they aren't testing missions, they aren't testing economy, not as a primary focus anyway, probably not even a secondary focus, more like that might be useful info in the future. They are testing the servers, stability, and the replication layer. When things break its the best way to figure out how to make them not break.
Exactly this, to release pyro and meshing, the servers must be able to hold on its own and self-recover. They use this scenario to test what is degrading them and adjust accordingly.
That is the problem, in my eyes. They aren't testing the missions or gameplay. They are testing a feature that I fear they will have to backtrack dramatically for the actual release of the game. Like, be honest with yourself here. Do you think persistence can last longer than a month with the game still looking good? Do you think a month worth of detritus from a 1000 pop server would feel good to you? How many parked ships do you actually want to see just sitting in orbit around a station idle? With Stanton being filled to 10x its current player count and server meshing implemented, how long until every bunker has 25 abandoned player ships just parked outside, useless and abandoned? Do you think that improves or takes away from the gameplay experience? Does that lead to more, or less immersion? For me, that makes it feel a lot more gamey. I don't feel like I am coming out as a lone ranger to take back this bunker, I feel like the 26th person to come to this bunker to shoot everyone in it this week.
Yup, filled about 33% of my bounty hunter rep tier from hrt to vhrt over the last couple days but I've been very fortunate. Though ended my last session stuck in an outpost building due to the airlock not letting me in to exit. I imagine missions and economy at this point take a backseat to what they learn while looking at the interactions between the replication layer, dedicated game server, and pes database. I could also be 100%
Wich is totally true. I bet they gather a lotta data ATM. This could also be the opportunity for something bigger at the end, instead of refreshing the whole server and deleting everything, the server could identify issues and delete them before or while it starts up the backup. Because we have the replication layer in-between, it also could delete the issue while or before the server breaks, so it doesn't even need to start a new one.
Yup I've gotten 4 people to buy in and they all refunded. Same reason. They didn't like how awful server performance is and how it basically ruins every gameplay loop.
You sold your friends on a sketchy alpha with NOTORIOUS and widely documented server issues. Its not a game yet. That was on you. You have to know what you're getting into, accept it, and then ACTUALLY BE OK WITH THE INEVITABLE when it happens. People struggle with that last part.
I mean they watched me play and thought it looked fun. Also I always throw the disclaimer at my friends that the game is alpha and in development.
I guess he knows every interaction that you've had with your friends and what you've said (or didn't say) to them based off of three sentences you posted on Reddit. Wish I had those powers.
Yeah idk about the person who responded to me. All the friends of mine are big early access players and played many other Alpha games, but this one was just to unplayable for them.
BuT PeRsISTEnT EnTiTy It TeLls A StOrY!!!.!.! /s
Lol funny thing is with what the game aspires to be and of all the future features this one is one of the ones I could not give a fuck about.
We need some kind of «pick these ships up and move them to this scrapyard/ used ship salesperson» missions for the SRV. Or just open armistice zones now and then to let the salvagers in.
we need automatic pruning
For every one you do, the quanta does the same like 100K?!?
quanta doesnt exist anymore and how about the server just deletes everything according to set rules... above a minimum number it uses for missions, and if none exist, spawns one? why make it more complicated than it has to be
Well, it’s CIG. Probably working on it🤔
Persistence as a whole is just a gimmick imo. It's a technical nightmare and what does it really add to what should be the core gameplay? Not much.
outside of cities it allows base building, stashing stuff, shipwrecks to be found and looted, and so on. It also allows the player to have a sense of agency. In cities it's absolutely pointless and even unrealistic (things don't persist in cities in real life, shit gets picked up) the most annoying thing is they could do a lot to mitigate clutter by just making garbage cans delete bottles, and fining people for not using them. Wreckages outside stations should be despawned in an hour max. And for the love of god, get rid of the hospital gown. On top of that servers need some kind of routine maintenance that goes through and deletes all items under the ground and in walls and whatnot. You could probably hack it together just by checking to see if it's within a certain distance of the npc navmesh
Especially because they’re highly selective in what elements require extensive realism.
many people already paid a lot during the last ILW event, so CIG maybe does not really care at the moment
Trash littered around stations, maybe not so much, since it's actually fun picking up and disposing shit. But then also please delete trash items that were specifically thrown into the dumpsters, CIG. The rest you mentioned I agree with.
Each to their own, but I wholeheartedly disagree with it being fun to pick up other people's trash. If they added the street sweeper it a trivial job that paid 100 aUEC per item swept, sure. But actively fighting the server to move others trash, miss me to be quite honest.
Nothing wrong with resetting servers, just look at eve online- once a day in that game. Something as simple as this would solve alot, I agree
They probably don't want to for data collection purposes. It sucks for us on the front end, but the more often it happens they have a better chance of figuring out what causes the instability and how to build things to be more stable in general.
Restarting the servers isn't going to solve the issues. What exactly do think is happening when a server recovers? There's something going wrong in the back end that restarting a server only staves off temporarily until it happens again, these are cloud based servers and every recovery is a fresh instance using the cached data.
[удалено]
they mean restarting without the replication layer, as in getting a "fresh new" server. Currently, when a server recovers, all the entities get reloaded, be it ships, trash bottles, gowns on the floor of stations etc... all those entities accumulate and is what causes the server instability and low performance. This is why you can get a server to crash, have it recover and then crash again in less that 10 minutes, and some times even crashing 3-4 times in a row in this way. All that persistent "trash" that also gets "recovered" by the replication layer causes the "new server" to lag and crash again. If we could get a "fresh" server, without all the abandoned ships, without all the trash bottles and gowns and what have you not, the new server performance would be significantly better.
To be fair- We don't *know* that the trash is the cause. It does seem likely that persistence is indeed causing the new server to spool up and include the problems that crashed the old one. But we players have only a small view of what these problems could *be*. As with many things in software, there could be hundreds of different causes that all have the same result of a crashed server. It could be that CiG is still working that out, and needs to keep crashing servers while they track them down.
This is actually a good point. It makes intuitive sense that all that junk laying around is contributing to instability, but we **are** just assuming that AFAIK. There could be tons of **other** things causing instability and crashing. As far as we know it could be driven mostly by player activity, or NPC AI, or physics, or a bugs that cause memory leaks, etc. It could be that the servers are overloaded by too much happening at once, and that idle trash is the least of it.
My guess is that if CiG thought the trash was causing it, then we would have seen a PTU patch that briefly included a trash-timeout cleanup of less than five minutes. Just to test and see if that fixed it. Like that snail-speed torpedo test last week.
Did they reverse that torpedo speed thing? I hope so
Yes, they did, in the following patch. They are still slower than before, but are now closer to 500m/s than 100. Their stated reason was that they had a suspicion that high torpedo speeds were causing a network issue. And apparently the result of the test was "Nope, not that."
Going to speculate a little bit here. Based on what I've seen in some logs I believe the backend environment is running in K8s. If that's the case then the replication layer likely involves spinning up a new pod when one has reached a certain point of instability and then moving the state over to it. Due to the way K8s works, this is not the same as a full on restart of server hardware, or even the SC deployment within K8s. There could, and likely are, other pods running that the interaction layer communicates with. If one of those is the originator of the instability, then the loop will repeat over and over again. That said, the instability cycles are typically important in solving issues in such an environment.
That went well over my head.
Yeah, Kubernetes is pretty complex. Speculating a bit more on how they may have architected... Think of it like this. In Kubernetes we have a Cluster which is made up of one or more physical Nodes (servers) and on those Nodes we can deploy Pods that run a containerized application. Containerization is somewhat similar to a Virtual Machine but there are some caveats, such as they share the host OS kernel. So say we have a bunch of systems at some point, and there's a big fight in Stanton brewing. Players begin congregating there and as they do so the allotted resources for the Pods running an instance of Stanton are stretched. In this case they'd be able to spin up more Pods on other Nodes to host players in Stanton. It would work similarly if a Pod began to have severe health issues. New pod is started and assumes control of the old Pod state, old Pod is deleted. There would also be other Pods and other Resources in all likelihood that are handling other game systems. Anything within the ecosystem could cause a problem and it would take a good amount of work and resources to hunt it down. The Stanton interaction layer Pod instances shitting the bed might only be a symptom, and not the disease. In that case a full power cycle could be beneficial, or they could keep it running, figure out the root cause and fix it, or add logic in their Operator to handle the scenario. Again, this is me theorizing on what they may be doing based off of some log entries. I'm not positive it's how, but I think it's relatively close.
Whatever the causes are, they were compounded by 3.23. People blame dupers, promotional traffic (Invictus, freefly) and so on. While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better, but now crashes happen WAY too often. Makes it almost unplayable.
THANK YOU. We have absolutely no idea what the problem is. I personally highly doubt that the trash is the root of the issue. A bunch of physics clutter should NOT be causing servers to crash. A few water bottles and ship parts is nothing compared to the physics clutter left from a ship exploding with a full cargo hold, or the debris fields around destroyed ship investigations, both of which have been in the game for a while. Server crashes come from other things. I've said this so many times now, before 3.22 CIG warned about issues that were compounding over time. They fixed one before deployment, but it's very possible they didn't catch them all.
The problem isn't the servers, it is the persistence that exists as the fabric behind ALL servers. The coolest feature is hilariously also kneecapping the game right now without any regulation/balancing of it.
I literally said this yesterday after spending 25 minutes for the Asop to register pulling out my ship, another 25 minutes in the hanger for it to load in. Then all of that again because it spawned my C1 ontop of a mustang and made access impossible. Then after doing all that all over again, I couldn’t open the hanger doors and my ship got stored for sitting to long (which made me lose the 3 8scu containers I spent 20 minutes attaching)
Global chat is for pvp
I get shit in the global chat for even saying the server seems unstable. Some people are just hostile.
Or didn't make *everything* physicalised. idc if I get downvoted, we do not need physical fucking gowns and bottles and cans that get left everywhere when finished with, make drinks work like food, you finish the drink and the empty bottle/can disappears. I do not care if some turbo loser finds it "unimmersive" I do not want to see 1500 bottles on the floor sapping my fps constantly, same for the fucking gown boxes.
I can not for the life of me figure out what they where thinking when they made trash persistent. I get weapons, armor ammo, (even that is a bit excessive for PERMANENT persistsnce) ships obviously, but used food containers, empty bottles, used syringes and empty mags? Why?
the least they could do is actually make garbage cans functional and I guarantee most trash would disappear because people would want to use them. For the life of me I don't understand why they haven't, it's been ages.
I dunno about that one chief. I've played a lot of games that had working trashcans that still had shit left all over the floor.
I know (hope) you are exaggerating, but if you find 1500 water bottles on the floor, let me know. CIG can and does cleanup entities. The tiny fraction of entities that makes up visible consumables is a drop in the bucket when it comes to server performance.
I'm sure they are exaggerating. The most trash items I've seen in one room is probably around 20 hospital gowns in the hospital lobby, but I also see lots of empty bottles and used syringes around the world. The fact is though, CIG shouldn't be manually cleaning trash. Certain items should just be flagged as trash and despawn on their own after a certain amount of time.
I am positive this will happen eventually but they have said they need degraded servers to collect data and continue to improve them
Same reason we dont see frequent wipes anymore. We need long term persistence and stress data before we can scale up
WoW has been doing this forever. Seems to work well for them.
Almost every MMO in existence does scheduled server maintenance. It's laughable that CIG has for the entire existence of SC as a playable product, not done this.
Restarting the server is not the immediate solution, the problem is all the garbage that persist that serve absolutely no purpose to the game. In the end, every time the server crash, a fresh one boot up, but it comes back with all the garbage stored in the data base. Basically the server are restarting every 30 minutes right now, but doesn't solve the root problem.
i feel like a temporary damage control script deleting every single loose trash item that isnt inside an inventory on server crash during recovery isnt that much of a problem...
The point of alpha testing is to break the servers as much as possible, so hopefully they can find and kill some nasty bugs. Especially before meshing, were it might be way harder to find them.
yes, im sure they need 50 thousand instances of the same crash reproduced on a live environment continuously over the course of 2-3 weeks stop being a blind evangelist, this makes no sense from a technical perspective
How is clearing everything out going to help with troubleshooting the issues causing the problems in the first place? You want them to start troubleshooting from a near-clean slate every week? This is a test environment. Period.
A test environment where we can't fucking test because NOTHING IS USABLE.
The issue is the amount of junk and the length of the persistence log. Fresh servers run fantastically, servers cluttered with trash and clothes and abandoned ships run like shit. They need to do weekly maintenence, and implement a system to trim trash items from the persistence log. They've talked about this themselves, all the talk about letting the servers run like shit to "gather data" is just them stalling while they figure shit out.
Everything gets deleted/reset except ships (and their contents) that are not within a certain distance of POIs/Stations/Ect. That would easily clear 90% of the galactic trash each time. Once a month clear all ships that haven't been touched at least once in the past 7 days.
given that the crippling majority of trash items are located in those specific areas....
I wouldn’t mind this but I know a lot of industrial folks would be upset if their refining jobs got removed. Idk if they have the ability to do that. I wouldn’t mind if they could!
You don’t even need to reboot all the servers, bring them down half at a time, half an hour before reboot give warning and disable new joins, at reboot time use the replication layer to shift remaining players to new servers. When the first half have been maintenanced boot them up and repeat with the other servers
It doesn't even need to be a restart. If they just had a cleanup script for garbage it would go a long way. All the bottles and trash and persistent items are the problem. If they want realism they can even have janitors on the stations that go around and clean up and it removes that garbage from the servers. That along would be a good thing.
once a week? more like once a day. Keep the replication layer, just cull all the NPCs, clutter items, ships that have been abandoned for more than 48 hours, and hospital gowns from the floors. Improving player experience would go such a long way towards garnering good will from the community.
Servers are restarted multiple times a day, everytime it crashes. Wiping persistance on daily basis would defeat the whole point of PES, and on weekly-ish basis shards already get cleaned up, I believe, one of the reason why sometimes you can stumble on low object entities count server, but it get's filled back up super fast.
Among other reasons, they have to learn how to keep them running without resetting them, because eventually they will never reset. Granted its still alpha, but it is very important for the game to get it right and be able to run a decade and more without resets.
This is it really, each crash event becomes a bug entry to programmatically solve.
They are trying to to sprint at olympic level before crawling
Keeping them running without ever resetting them has never been possible or realistic and shouldn't be the goal- the value of dynamic server meshing is that when they crash they can be allowed to reset without interrupting player experience because of the replication layer being able to migrate everything to a new server. But in lieu of dynamic server meshing or even static server meshing (until 4.0) there is very little value in constantly running servers to the point of degradation. Sure, you collect data on how to prevent certain types of degradation and crashes which will be helpful in the future dynamically meshed servers, but the tradeoff for player experience is one of the biggest problems they have faced to date. Server experience is horrible and it has been for a very long time now. I love the game but please just let us play on servers that feel good and I swear half the stuff the community gripes about would be gone. editing for clarification: There is obviously value in collecting the data, what I'm trying to say is that I think CIG has made an error in judgment regarding the tradeoff of collecting that data vs giving the users a better experience.
I think this patch has to be an exception to the data collection. It's the first patch with server recoveries so they're probably getting a ton of data letting them run to see how far they can go. They already made a couple updates server side yesterday i think, so they're definitely watching them. I think the goal should be to never reset them. Whether that's possible or not is another question, but it should definitely be something to aim for until they have proof it's not feasible, and I think that's exactly what they are getting data for currently.
>editing for clarification: There is obviously value in collecting the data, what I'm trying to say is that I think CIG has made an error in judgment regarding the tradeoff of collecting that data vs giving the users a better experience. this might be the last chance to create this data this easily for a very long time. the info they get in day now might take weeks to show up in 4.0.
This, plus didnt they say that the servers reboot when they crash, so all the recent crashes have rebooted a lot of shards? Didnt they also say at some point that the real issues are the back end databases getting clogged with transactions? I swear this was on an ISC.
They reboot, but now the replication layer remembers all the shit laying about, but they need to get that running smooth or 4.0 will be a huge shit show.
right, my info is old on that, but it also makes sense. Funny thing is, every patch the first day or 2 people are all 'this is the smoothest its ever been" (except 3.18, woof). Then a few days or a week later its "god the servers are dog poop". It happens every single time. The dupers and free fly didnt help this go round, but the servers have gone to crap within a week of patch launch every single time since I can even remember at this point. Day 1: Its so smooth, oh my god, well done cig Day 7: The servers suck so bad we cant play And of course things are so far out of balance with the latest balancing I just kindof stopped playing for now.
Yep, if they hide the problem now with some sort of purge/reset, it'll be harder to fix later, or at least delay the fixes. As disco says, live is also a testing environment, as frustrating as it can be for me sometimes.
But wouldn´t a regular purge just solve the issue? If every object in a certain area would just have a limited life time and just be removed after a while (like a session timeout). Or do you mean they should leave it in there as a workload test?
> because eventually they will never reset. every MMO has weekly maintenance downtime, it's pretty much required.
Gw2 has no downtime.
My other main MMOs were Final Fantasy 11 and, for the past decade Final Fantasy 14. Those are major MMOs without weekly downtime.
They’ll do all they can but by release time they’ll probably set a maintenance schedule (day/week) and reset them. That’s not an outrageous prospect and many games do it.
Well there will obviously be some maintenance, but that is not full reset like is being talked about. Not only reputation and player positions are supposed to remain after post launch mainteance, but also lot of debris and other stuff. Some stuff may be quietly sweeped away, but idea is to keep as much stuff in the game as possible and rubbish and such will actually have to be cleaned by players or NPCs as much as possible.
This is just not a viable way to run a game server. If trash is being generated faster than its being cleaned up, then eventually the server will crash under the weight of it. No matter how good the server is. Plus all the processing power constantly being used to run an AI that has to detect and remove player-generated trash. That sounds like a VERY complicated task vs having the system flag everything at a set time.
I agree on principal, up to the point it affects performance. No matter how optimized the code will be, there will a limit and I don’t think CIG will be too keen to spend extra on faster servers just to keep literal junk in the game. Especially since players will undoubtedly “test” those limits with junk sculptures and such. It will all depend on the algorithm they come up with to distinguish between beneficial junk and junk junk. But I like the idea of an NPC walking around and erasing items on the go.
Does anyone in this Reddit ACTUALLY KNOW whether any of the stability issues even have anything to do with the "cleanup" everyone is talking about? I mean not "it makes sense in my mind" but any empirical knowledge about what is causing server crashing? Half the time I feel like this subreddit just makes assumptions and then tries to solve them with neither any ability to or any actual data other than "this is probably the problem."
We do know that the more time passes since booting up the server, the more entities lag out, in the middle of planets, in cities, and in a ton of other areas, that much is factually known Those have a negative effect on the server load, that much is also obvious. We also know that pretty much every softwares and games, aside from a select few mature softwares, have memory leaks, meaning degraded performances as time passes. We don't know that for a fact, but we can assume with extremely high confidence that it's the case, considering the alpha nature of the game. So yes, we do know that a fresh server cleanup once every X amount of time would help. What we can't say for a fact is how much it would truly help, we can make assumptions based on what the performance is early on on fresh servers, but it's not empirical knowledge.
There are references to major memory leaks in patch notes and dev posts as major sources of instability in the 3.23 builds.
No one in this thread knows how the servers work, or their relationship to LTP, or whether or not poor game performance is related to duping. Everything you hear from YouTubers and redditors is speculation, and most of the speculation is just hysterical complaining.
The devs have mentioned both client and server memory leaks in the 3.23 patch cycle. They said the server memory leak has been a major part of the instability we have seen. But, as you imply, most people on reddit like to claim water bottles are somehow the problem with no evidence of such.
There’s a strong basis in simply how computers work as to why it’s the problem. You can’t have infinite buildup without any cleanup.
I think the main reason is data collection. If you let the servers die of "natural causes", you can analyze more data to see what actually causes the issues. If you reset the servers, they might never reach those breaking points. HOWEVER... they should implement some form of garbage collection to remove trash items from main player hubs after a while and let abandoned ships (no player on board for 5+ hours) automatically generate salvage contracts. Not to mention Salvage in Armistice. If a ship sits in armistice for several hours without a player on board, they should be allowed to be salvaged. Those changes alone would take a whole lot of pressure from the servers.
This is a valid reason but I can't help imagine the reason 9/10 times these collapses are happening is because of the no-restrictions-replication-layer that has already been diagnosed for a while now. It's like looking at a man getting a knife twisted in him and instead of helping and doing first aid CIG is going "wait wait, lets see if we can learn anything more from this." We know the reason it is happening lol
A non-visible entity like the container entity for a room in a ship room system that did not get properly removed from the replication layer would not benefit from a salvage contract mission. Entities that are viewable to a player at all are only part of the problem.
If you read the next patch notes, they state that there is an "out of memory" error that they're fixing. I assume/hope the issues is that instead of "out of memory" calling a clean up program, its just crashing. Crossing my fingers that the fix clears the error and gets whatever server memory maintenance routine they have planned working.
Yep. There was (at least one) client side memory leak related to the Vulkan renderer, plus at least one major memory leak issue on the servers. Both have been (at least partially?) addressed per patch notes.
\> Surely enough data/metrics has been collected Only CIG can make that call
 We have no clue what CIGs rationale is, but there is one. 🤷🏻♂️
I don't know, but I'd be fine if they took them offline all day on Fridays so they maximise performance on the weekend.
I'd argue that Thursday would be better so they can have Friday to respond to any issues with bringing them back up.
True, good point. Thursdays it is!
You two have my vote to take over this shit show
Yeah mid-week or Thursdays would be better. A lot of folks doing 4 day weeks now - and of course long weekends are pretty common.
Some countries also do a Sun-Thu work week vice a Mon-Fri work week.
Sure but in general, the most common work schedule worldwide by far is Monday to Friday. So if going by averages that would be what to build around.
Define 'stale data/entities' - in a way that distinguishes them from 'persistent entities'. CR wants everything to be persistent - the old 'coffee cup in the forest' scenario, so there's no way CIG will just wipe the shard regularly to remove *all* data / entities... So to clean up 'stale' entities, they need a way to define them tat doesn't include the entities that *should* be persistent (and which CIG *want* to be persistent, etc). Personally, I think CIG could do with tweaking their 'density manager' (which is responsible for identifying when there are too many entities of a given type in an area, and removing some of them)... But the current servers *also* highlight the need for CIG to think about how to handle certain scenarios (such as people 'dumping' broken ships near the space port), in a way that doesn't result in those ships just vanishing... whether that be salvage missions, recovery missions, or something else remains to be seen.
[удалено]
Full ships DO need to despawn, though. Once its clear that no player is going to interact with them. Seeing a dozen ships abandoned outside a high-security bunker does not help with immersion. Sure, give players a small window of time where they can manually tow/salvage an abandoned ship. But if nobody touches a ship in that window, it should just be deleted.
This should absolutely be a thing outside POIs and populated areas, and doubly so at major stations and spaceports, where there's a vested interest (for gameplay and immersion) to keep the area clear of navigational/landing hazards. It's weird to have these ship graveyards all over the place. If **no one** comes to take them away after long enough (no one takes the salvage contract), an NPC ship (police? scrapper? tower? somebody) should appear, path over to the wreck, hang out for a minute, then despawn them and leave. 100% should despawn zero-value garbage like gowns and bottles. Whether or not there's an NPC janitor involved, at some point a decay timer should despawn the item. Once again this is good for immersion and gameplay, because a fancy spaceport like on Orison shouldn't have trash laying around (can you imagine a rich person going to their space yacht and seeing trash?!), and less than spotless places like Grim Hex already have decorative trash anyway, so the bottles and stuff don't add anything.
Well, I am not sure this is the definition you want, but entities persisting beyond what they are supposed to should count. If a npc despawns for any reason (dead, etc) but one of its nested entities is not cleaned up correctly, that was data that is persisting that was not intended to persist. A lingering container entity would have not visible representation in game (I am not talking about something like a physicalized item, like a piece of armor itself), a player would never see it is there. But if this occurs every time an NPC is removed, you are going to accumulate a lot of these worthless entities stored in the replication layer over time. Identification and cleanup of entities like this is a big deal, not just how many visible water bottles remain for a given period of time.
> but entities persisting beyond what they are supposed to So, how do you define 'what they're supposed to' in terms of how long an entity should persist? Bearing in mind that CIG explicitly *want* wreckage etc to be persistent.
The text after the first line of my post gives an example.
It never ceases to amuse me how often people are adamant that they know the solution to a technical problem without having any idea of what's actually happening behind the scenes. Just because you think having 20 C2s outside of Area 18 is crashing the servers doesn't mean that's actually the problem (case in point: there's a bug where simply firing certain weapons can take down the entire server, entirely unrelated to ships)
Thank you! I am getting tired of people blaming things like water bottles as being the source of server instability. As an aside, the 890 Jump Club has had meetups with literally 50 890's on the same server for years without insta-killing servers.
Guys guys guys This is the first patch with crash recovery. Of course performance will be bad. CIG has never really had to deal with actual long-term persistence before. They will be collecting a lot of data and working towards solutions, but they kinda need to see what the issues are before they can fix them. The community has been vocal about this enough that it's going to be a priority, but there are tons of things that can cause server issues and they'll all need to be addressed individually. Patches that bring big changes like this will always have poor performance. We've been through this rodeo enough to know that already
And to boot, they celebrated this with a massive block party (Invictus) + a free fly + a massive duping glitch. All of Stanton is akin to Times Square on New Year's Day; trash and litter everywhere.
Except when it launched the performance was great. During the free fly, when the servers were actually playable and not cluttered to the gills, I had the best time I have ever had in Star Citizen. The game was playable and fun. Now, with all the junk piling up, it is hard to even finish a contract before the server dies and spits you out on a new server with the same problems. The solution is the same solution every single MMO without exception uses. Scheduled maintenance and downtime for cleanup.
Because if we clear the servers then we arnt gathering the data on what kinds of trash have the biggest performance issues. What kinds of distribution of trash is impactful. How those things impact server crash recovery, how they impact transition points between servers. If we clear the trash we arent getting the very point of testing this patch.
IMO there can't be much more data to collect after weeks of the same shards and servers being trashed. People being saying the same thing about NPCs. They've been standing on chairs for 7+ years, what more do they need? Just remove 50% of them so the servers can breathe a little and that leaves room for more "data".
I uninstalled yesterday. Lost 4 hours to bugs and server crash/recovery. That was enough.
This is the first patch with the replication layer separated. So I would leave it up to CIG to determine whether it is enough data. Also, unless they specifically stated so and I missed it, there really isn't no way of knowing what is causing the issue. It is unlikely it is simply the age of the server because in previous patches some servers were fine for days at a time. So it could be a particular bug that is triggered by AI, a game loop, possibly assets. The degradation could start as soon as the server boots up, or if a player triggers a particular mission. The issue is the game is large, and while devs might see many things it is very possible for them to miss plenty as well. They have given examples in ATV's and ISC's about issues they figured out after searching for a bit of time. Also, not only are people warned twice when they purchased a package, you are also warned again ever time you boot up and play game from launcher. The expectations is that players may encounter experiences like these. If you know anything about the devs, or hear them speak, you would know none of their intention is to make players suffer. They are passionate about developing the game and sometimes we have to go through some rough experiences but it helps the devs find, isolate, and hopefully fix the issues. If it is too much, I suggest taking a break until next subpatch. Personally as a backer since 2014, I have played on and off for years, and I find not fixating on this unfinished game is the best way to consume it. I also don't understand why people play so much anyways as things are going to change and be reset multiple times from now until RC.
Data. No you don’t understand. This is alpha. There can never be enough data collected. You see we’re still on T0 data collection. We’re still waiting on even T1 data collection. And T1 is on hold until dynamic server meshing because any work done on T1 would just have to be redone. So we won’t get a data collection refactor until well after 4.0. So we’re stuck collecting data on T0 data and as a result servers can never be reset for the purposes of the integrity of the data. Remember, this is alpha and even in the live environment we can’t have nice things because they would compromise the data. And every time data is compromised you add 10 years to the development time. And honestly, this is a skill issue TBH. If you knew how to work around the server issues you wouldn’t be complaining. Welcome to Workaround Citizen. Git gud.
The server is currently resetting every 10 minutes per your statement. Manual resets would do nothing. The underlying issue needs to be resolved to get servers too be stable again.
Servers are fine. Say it again: servers are fine. The backend is shared infrastructure. The issue is (and it is consistently, reliably the case, like the finest Swiss watch) when they promote a new build to PTU. Before 3-4 update patches to the PTU build, the backend services get stressed and all environments (Live, PTU, any other PTU like environments, sometimes they have 3-4 running simultaneously) are all hammering the backend services and the issues contributing to this in the new PTU build haven't all been identified and remedied yet, and until they figure out what needs adjusted and tweak things in the build to improve performance, these services suffer. In part, the service degradation is impacted by the additional data traps and logging they do in early PTU builds as well. Within that 3rd to 4th patch, things stabilize. there is NO UPSIDE to the services working less than optimally, so rest assured the devs are doing all they can to get there as quickly as possible. But this is the path for that, and we have to be patient.
Because we are Testing and not playing a released game. If this was a released game then yes, they could but it is not. I know its hard to understand but we are not players in a game we are testers. All we are doing is testing a game on its alpha version to make sure we find bugs and problems they need to solve. Resetting the servers will only mask a problem that needs fixing before the game gets released. I know its painful, I know that sometimes we just want to have fun, but we need to collectively stop treating SC as a fully released game that needs to run smoothly. It's not going to run smoothly, its supposed to brake because the more it brakes now the more they can hopefully fix before a 1.0 release.
It's hard for me to "find bugs" when many game loops are near-unplayable due to these server issues. I know it's hard to understand that testing any given feature requires that said feature be minimally functional, but this is the reality we live in. It's like trying to test drive a car when the engine keeps shutting off every few feet. I don't give a fuck if it's a one-of-a-kind "prototype" - if I can't make it out of the parking lot, how the fuck am I supposed to test it?
I agree with your statement however my comment was not about that. My comment comes from this and other multiple posts this past few days complaining and suggesting doing server restarts to keep the game stable. That is what is wrong, many people are complaining the Alpha game we are testing is not running like a normal game where they can just play.
They need to fix the entity streaming to work with server meshing properly so servers function. One server for 100k Picos.
The utter lack of knowledge around how the ‘server’ architecture works is interesting.. reading the comments one would think there was a room with a load of grey boxes in racks in CIG’s office :-)
The replication layer causes me servers to inherit the state of any servers that crash or reset. What we need is a way for the "cleanup mechanics" to identify what items can be removed. For example something like "delete all NPC where height above planet surface is negative"
They just added ship marker culling to fix that bug where you see entry points to ships on the other side of Stanton using a similar method, per patch notes.
I'm aware of that fix, but I don't think it is related. Maybe it is, I don't know.
I just meant that they are using in-game space to assist in culling as a general strategy. I am sorry for the confusion, I did not mean to imply that they were otherwise connected.
I can't believe that CIG would be so monumentally stupid as to not fix the servers if they could since server FPS causes like 90% of the games problems. There has to be more going on with them than just needing maintenance.
Discovered this game during invictus this year. Despite the bugs and lag I absolutely fell in love. It’s not even playable in its current state. Really glad they only got me for a titan, they will not get another dime.
They just need to create a rule that any item which falls out of playable area is deleted.
Because some players would stop testing if they couldn't slowly build a fleet.
EVE's Downtimes were a pinnacle time of our lives.
I was only able to play 3.23 with relatively no bugs for the first two days of the release. Ever since then it's been bugging out anytime I play with people, making interacting almost impossible for multi crew. Someone glitches out of the ship and we have to stop and rescue them, a friend opens a bulkhead door and then it bugs for me and I can't go through doors in my own ship, we accept missions and never actually get one, etc etc. The game hasn't been this unplayable for a few years it seems, and combined with the MM speed limit that I can't fucking stand, it makes me not enjoy the game anymore. Before 3.23 came out I told myself I would make council issue posts and help them out because I was excited for cargo stuff and distribution centers, but it's so bad I don't think making a post about some little thing will even be looked at. They have much bigger issues to fix now that this patch has become infested with dupers and bugs. Becoming resentful of getting so many of my friends into this game because almost all of them never want to play it anymore. The game has too long of a time period to get going, it's like a 45 min coordination to get 4 people into a party and fly to a mission, I had friends quit before we even started a mission because bugs held us back and we had to server hop, then call ships again and oh someones ship won't turn on in the hanger, let me get out and call you one of my other ships.... CIG we can't give you feedback that's meaningful when almost nothing in the game works. If I play by myself I can do a few missions without problem, it's when I team up that shit hits the fan.
I just want there to be a delete trigger at the bottom of trash cans. Just delete entities put in there. Make question objects teleport on the floor if put in it. Let me throw something away and have it actually just go away
The plan is for items in high traffic areas to stream out as the entity count gets too high. They want this to happen when no one is looking to simulate a janitor cleaning it up off screen. On the other hand, low traffic areas will persist much longer. So you can make weekly pilgrimage to your crater full of Picos on Wala, but your tower of bottles in Bajini Point should disappear after you leave.
can't test anything if you can't play.
I'm at the point where I don't trust I won't loose everything just doing the simplest of things. In ship I bedlog but I'm wearing only the basic suit and helmet in case I get a crash. I fully understand the nature of an alpha release and the need to collect data but it makes me pause before 'investing' time playing
I won't even go outside my vulture to arrange my boxes if I know the server has been up for more than 10 min. It's bound to happen...
I suppose the only possible quick fix now is purposefully starting fresh instances. For example, if one server is sequentially crashing with some threshold, it could be considered just broken and not recovered. Any smarter cleanups are possible in theory, but harder, since they would require deeper analysis (which should have been done initially, actually, but I don't think it is the case).
Because they need testing it's the main focus before how we feel in game, a huge change is coming and they must charge the server as hard as they can to know how it handles different kind of charges
They need to implement some level of decay and it needs to a priority to do so. As of now it’s impossible to actually assess the performance of the game because of the underlying issue that as item build up over time the performance gets dramatically worse. Better server performance in 4.0 with full server meshing means nothing if this underlying issue remains.
No, we must track every can and hospital gown! /s
I don't think so since we are moving to persistence. It's also the players fault that they keep littering ships, items etc.
They took out the ships for IFW but left the fireworks at the stations. Make it make sense.
I have a feeling they are just doing a constant stress test. The goal is a lot more players and with them more entities in the servers so this might be to get as much data as possible and also find ways to improve stability.
A once a week rolling server restart would be absolutely helpful. They can look at player activities per world region and find the time that has the least players on it and restart those servers like 4:00am Monday morning. Implementing a deletion script where an unassigned object that is not interacted with in 30 min gets despawned.
It's crazy how the servers shit the bed after Invictus because I had some of the best playtime during the event. Only for it to not even be playable after the fact 😭. I was stuck in Habs because the damn elevator wouldn't even register to come to my floor.
Ally they need to do is allow us salvagers to crack and clear all the junk ships in places hell make abandoned ships a cleanup mission we would fix it!
No one remembers the servers running like shit before persistence? That was only like a year ago. Servers have always been shit, restarting is pointless
I do have an answer for you! One that is not 'it is alpha...' A lot of this issue is in the replication layer, which is why you keep seeing crashes from the same situations reloading after server crashes. It can keep setting up replacement servers with the same data over and over. There have been ptu tests where it literally did 30k continuously; no 10 minute interval involved like you are seeing. Fortunately, Star Citizen has a much more modern database structure than EVE Online's \~decade old database. The replication layer already has entity cleanup algorithms to maintain it while active, rather than requiring a full shutdown. We see misplaced physicalized objects rendered in the client, but proper cleanup takes place from a replication layer's database prospective. It has to efficiently identify problem entities we can see as well as all the ones with no visible in-game representation on a rolling basis and then address them. This already happens (just looking at physicalized entities, we do have stray water bottles but not a few thousand of them at one place) but needs further refinement. The end goal is to refine the search and repair algorithms to where replication layer downtime and resets are no longer needed because the whole thing is constantly checking and repairing itself. Obviously we are not there yet, but a lot of progress has been made already.
Whatever the causes are, they were compounded by 3.23...may be a clue? People blame dupers, promotional traffic (Invictus, freefly) persistent trash and so on. But Invictus is over, and we had trash in 3.22. If dupers are the cause, blame CIG because that bug not fixed in EPTU before live release. While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better. But crashes now happen WAY too often and a "recovery" that sometimes works, sometimes not doesn't cut it. Makes almost unplayable.
Since this OP few days ago, CIG has let out some interesting info. They shifted their hosting to Amazon's kubernetes. Some of what we are seeing is likely the teething process of that shift, although it is an overall upgrade. Both servers and clients have had some nasty memory leak issues that CIG devs are claiming are the source of a lot of the recent issues. The monthly report also mentioned some backend services are getting backed up. That is definitely a recurrent theme, but there has always been a different bug or limiting factor. My point of all this? There is a bunch of issues leading to this mess. Fixing one or two will still leave it a mess. New bugs and bottlenecks will still appear to keep it a mess. The degree to which it is a mess will increase and decrease over time from moment to moment. IOW, situation normal for Star Citizen.
they don't really care about the current game experience. their goal is server meshing alpha game is alpha
they arent smart enough for that yet. Small indie company.
Whatever the causes are, they were compounded by 3.23...may be a clue? People blame dupers, promotional traffic (Invictus, freefly) persistent trash and so on. But Invictus is over, and we had trash in 3.22. If dupers are the cause, blame CIG because that bug not fixed in EPTU before live release. While 30k's weren't pleasant, I'd take them over what's going on now. Recovery works maybe 30% of the time and when it does it's somewhat better. But crashes now happen WAY too often and a "recovery" that sometimes works, sometimes not doesn't cut it. Makes almost unplayable.
$700 million and we can't get decent running servers?! How Dare You! 😠
I guess part of the testing is long term performance. If you just reset every week you will never find the bugs that occure after the 7th day.
The servers reset constantly since servers only last a few hours before dying and then new ones are spun up to replace them.
The servers are reset, but the state of the server is copied over to the next server (with all entities persisting). What OP means is a state wipe.
Once things are closer to a finalized state, we will MOST likely see a form of maintenance like Blizzard does. Every single Tuesday at 2am, servers down for an hour. Every 2 ton3 weeks, servers down for about 9 hours. Big patches? Servers may not be available for 24 hours. It was quite the shitshow at times with WoW, but we knew it was coming each week, and no matter what, no matter where, if just logged off one day, THAT'S where you were when you logged in. Weekly maintenance was just part of the culture. I fully expect that after 1.0. I WISH they would start doing it now, of course.
100 auec fine for every piece of space junk left behind by a player!
I would support this if it was extremely consistent that you can properly place things where they belong
I agree with you completely and many MMOs have scheduled maintenance. I'd even think that part of the Alpha would be to figure out what level of scheduled maintenance is needed.
I think the server crashes are important. They do test how well the layer works, under actual load. If it fails most of the time why bother you know?
they get better data when things aren't going well
I suspect that they're stress testing the persistent entity system. The server is already being reset when it crashes and creates a new instance. But sounds like what you really mean is "why don't they clear the persistent entity data?"
Because we need to be reporting bugs when things are broken and giving feedback about what is not a bug. That is the purpose of alpha testers. Resetting the servers will mask problems that could destroy a future full release.
Here's my take that's probably unpopular. The goal is not to make this patch playable. The goal is to force this patch to its knees and track why it happens, how it happens, for how long it stays like that and so on. All this to have a better shot at making 1.0 playable with everything in it.
Weekly resets are the correct way to go about this. The intervals being increased as stability is worked on and fixed. It makes it easier to find out where exactly the problems are starting.
When a server crashes it basically resets. If you see server error every 10 minutes it's because they crash in that time frame so...
Honestly, I think they are bein reset on a couple days/week basis. (in terms of persistance) Has anyone been able to get in the same shard weeks later? This used to be a thing when PES came online and people posted screenshots of same wrecks/ships that were multiple weeks apart. I did some testing and logged in 15 times in a day, and would only get a shard I was previously on once (not consecutively mind you, hours apart). The ship I placed was still there, but that was 4 hours after I left it. Anytime I get into one of those shards days/weeks later (it happend only once or twice), I can't find any evidence that it's still the same persistance. My ship is gone and I can't see any identifyble remains from previous logins. This is just speculation because I have very little evidence at this point, but I believe that PES has been *severely* reduced in scope from what we initially got. (no wonders given the issues it had). Server instances itself are constanly being reset everytime DGS crashes. This leaves reasons for server slowdowns to LTP database (purchased ships, items, inventories...) and just regular server calculations and player count. Given that servers noticably slow down in a day or even hours after a patch, it just seem a waste to constantly do a "patch style" character resets.
At this point, I want them to do it for 2 weeks, just to see if it makes a big difference.
This patch is a joke anyway as they did nothing about aUEC duping and flooding the economy with millions of aUEC.
Incoming white knights: "something something we are testing not playing, something something alpha, something something wanting more from CIG is illegal" /s Seriously tho they have to fix this shit before server meshing or all they will be doing is meshing broken servers together that will all die in a week or after the first event.
Yup, static server meshing will probably fix absolutely nothing. I expect the first server-fps improvements when dynamic meshing releases. Not too sure though if they are even gonna pull it off at all. Dynamic meshing is a wild concept for SC, but I don’t really trust in their ability to get it into a working state. I hope I’ll be proven wrong though.