Grid Issues: Old IMs Reappearing in Chat

We have had reports of old IMs, some as old as 2 months, reappearing in chat as if they were just resent. We want to assure residents we believe this is a technical problem and not associated with anything intentional or malicious. We are investigating the cause and if you wish to send in a support ticket with your experience we will pass that information on to the investigative team.

Posted on Wednesday, December 9th, 2009 at 6:11 AM on Grid Status Report.

This bit of news was already known to some, judging by the chatter in some groups, "You mean they just noticed that?" One lady claimed to have gotten someone's reply to a question a little late, five months late.

moreless

Stability, a real boon

It is funny how one thing has changed over the last year. Not so long ago, you would hardly see 40000 people online, perhaps on a good day. Nowadays, it is common to see from 60000 to 80000 people online on a regular basis.

What happened? It looks like the Lindens' efforts to make the Second Life grid more stable have paid off. Gone are the days when you would have to wrestle with your client and connection just to log in. Unexpected grid crashes, unplanned restarts and downtime are now few and far between.

It was annoying for grid residents to deal with a grid that wouldn't let them get in or would kick them out at any time because it couldn't withstand large-scale traffic. This made people delay projects, lose sales, and miss events.

Stability is a real boon but let it be clear that all challenges did not find their solutions. If SL wants to "make it big", it has to be adapted for mass-scale use.

In RL, you can get thousands of people in a sports stadium. The only real limit to the number of people you can squeeze somewhere is actual size. In SL, a sim usually slows down when 40 to 50 people are around. Avatar rendering costs play a role in this, so do textures and "physical" objects that interact the everything around them.

People with more technological knowledge than me would say that SL has to be scalable. If works nicely when few users are around one particular place. But if you try to squeeze 100 avatars or more around a sim, the experience is atrocious.

This might even be the biggest challenge for Linden Lab. You can make the grid as huge as you want, but if you cannot concentrate avatars in one place for a special event, you lose much appeal. And we have yet to see real progress towards that.

moreless

Weekend Grid Outages

Although the second half of 2008 showed a big reduction in usage hours lost to outages (we reduced outage hours by over 50%), stability challenges have increased over the past month on the Grid. This weekend was especially painful, and the first time since joining Linden Lab that I’ve experienced a full mySQL crash (this occurred just after 4pm PT on Sunday).

When the central database crashes, it takes approximately 1 hour to rebuild tables and indexes before accepting queries and becoming fully functional. This was the main reason for us to employ the painful triage process of temporarily blocking logins, while the database is in an overload state. These 5-10 minute “blocking periods” are substantially less Resident impacting than a full database crash, and a 50-60 minute restart cycle. However, neither is acceptable and I wanted to continue updating our efforts to stabilize the infrastructure.

You can read the rest of what Frank Ambrose (FJ Linden) has to say HERE

But I personally would say a great deal more is wrong than shows on the surface? How many of you can get to the support page from the secondlife website? or even from individual articles - I had to find a backdoor through the wikipedia to be able to submit a support ticket.
How many have had their credit card suddenly rejected after a long time using the same one?
Some have shown signs of these problems and these are on top of the actual inworld issues which are plaguing us all
I for one wonder what exactly is going on at Linden Lab just now as they seems determined to drive its most staunch Residents out.

moreless

Post Christmas Grid blues

Each year, after the Christmas gifts have been unwrapped and all those people who treated themselves or family to new computers/Internet access, Secondlife sees a surge of new signups and more people using it than ever - the problem with that?
Quite simply the grid cannot handle it!
Most of us who spend regular time inworld are aware of the issues and mostly what is behind it - basically once the grid reaches 75k+ online everything goes down the toilet - teleports, transactions, rezzing, building even dressing becomes problematic.

Lets have a look at the last few days:

[RESOLVED] Logins and In-world Issues

Posted by Status Desk on January 6th, 2009 at 10:45 am PST

[Resolved 10:45am PST] - The database has recovered and all issues have been resolved.

[10:30am PST] - We are currently experiencing a problem with the database. There may be difficulties with logging in, rezzing objects, completing transactions, and searching. We are aware of this issue and our ops team are working to resolve it as swiftly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects.

[RESOLVED] Inworld Issues/Logins Disabled
Posted by Status Desk on January 6th, 2009 at 04:10 pm PST
[RESOLVED] Inworld Issues
Posted by Status Desk on January 7th, 2009 at 02:58 pm PST
[RESOLVED] Inworld Issues
Posted by Status Desk on January 8th, 2009 at 01:34 pm PST
[RESOLVED] Logins Disabled
Posted by Status Desk on January 10th, 2009 at 01:54 pm PST
[RESOLVED] Logins Disabled and Inworld Issues
Posted by Status Desk on January 10th, 2009 at 04:53 pm PST
[RESOVLED] Logins disabled
Posted by Status Desk on January 11th, 2009 at 11:20 am PST
[RESOLVED] Logins disabled
Posted by Status Desk on January 13th, 2009 at 03:39 pm PST

Each of these represents times when Logins have been restricted, a black week for any Secondlife business.
Some have called for allowing only premium accounts to log in at these times and yet others slam this notion as many who would like to go premium cannot do so, therefore making the idea unjust.
Then there is the question of bot activity, if as Linden Lab numbers suggest, 10% of those signed in are bots then surely shutting these out would allow real players more of a chance to log in?
But how do we or Linden Lab know which accounts are bots?
Age old question and one many have tried to answer - but to be honest our answers do not matter Linden Lab will do what Linden Lab chooses as always.
I am not technically minded but even I can see the grid is struggling - Linden Lab says 10% bots - most believe that is a very conservative estimate it is more likely to be around 25 - 30%. Now how about if the bots were registered? Then when the Grid is struggling bots are logged out? Wouldn't that make more sense than locking out paying Residents? Even those who run bots would rather have paying Residents inworld than their bots surely?
Hopefully with all the grid issues Linden Lab will finally make a move on the traffic scam that causes most to use bots - once the reason for them is gone it will make the need for bots redundant. But for now Linden Lab have chosen to chase their tails and deal with issues as they arise rather than dealing with the root cause (as always).

Dana Vanmoer

moreless

[Resolved] Land Store, Logins and In-World disruptions. (RESOLVED - honest)

Amid many celebrations and anniversaries this weekend is the Grid status page has left many thinking the problems teleporting, transaction failures and basic lag issues are only a problem for them;

[Resolved 12:00pm PST] - The Land Store has been re-opened and is functioning as normal once again.
[Update 11:35am PST] - All services have been resolved and are working as normal again, with the exception of the Land Store which will remain closed while we test functionality. We will update this blog when the Land Store has been re-opened.
[9:28am PST] We are looking into issues affecting transactions, logins, account creation and possible extensive related disruptions of services. Please refrain from any transactions as possible while we work to resolve this issue. During this time the Land Store has been closed to avoid further complications.

Guess what guys it is not RESOLVED I have spoken to many people since this was published as a resolved issue and believe me it UNRESOLVED but of course we have no way of letting Linden lab know this since we still cannot comment on the grid status page
Lets hope it actually becomes resolved before our own events begin tomorrow!
Dana Vanmoer

moreless

Frank Ambrose (FJ Linden) introduces himself on the Blog:

"Hello, I’m Frank Ambrose, the Senior VP of Global Technology, and I’d like to take this opportunity to let you know about some of the work we’re doing on the Second Life Grid.
By way of introduction, I’m a recent hire here at the Lab, having joined to lead our global technology team. Specifically I’ll be focused on grid infrastructure and our stability initiatives. As noted in the press release, I come to the Lab from many years at AOL (and prior to that MCI), where I experienced the kind of explosive growth, global scale and inherent stability challenges we face here at Linden Lab.
More than anything else, my tenures at those companies taught me the direct relationship between platform stability and user experience. I’m looking forward to applying that lesson, and a host of others, as we work to maintain, build and improve this complex virtual world. I am keenly aware of the pain that any service outage can cause and am both excited and confident that Linden Lab has focused the right resources to achieve this critical objective.

Given the complexities in our architecture, our stability efforts span many individual areas, most of which were detailed by Ian Linden’s May posting. Some areas will be addressed through short-term initiatives, while others will require significant re-architecture, software changes and new physical hardware. Throughout it all, we’re committed to making the transition to a more stable world as seamless and transparent to you as possible. To that end, members of my team will be using the blog regularly to provide updates on plans and progress towards meeting our stability goals."

Read the full introduction HERE

The main point seems to be that Linden Lab are working on stability - but we have all heard that before, right? So let's sit and watch the fun begin as the recent, fairly stable, grid gets pulled apart in the hopes of finding a long term solution.

"One of my initial observations is that many components of the infrastructure are “over engineered.” That’s not a criticism, its just a fact.

There are a lot of reasons for this and we’ve already taken steps to remove some complicated code that was deployed and contributed to some of our recent instability. It was designed with the right intentions, but just didn’t deliver.
Finally, I want to restate that we are committed to fix the grid. It’s my singular focus. But I also know that words are cheap and we need to start delivering on these promises.

If my days at AOL taught me anything, its to have a thick skin, and sense of humor. Whatever the comments may be, I’d much rather have an active group of passionate residents (frustrated as they may be), than a silent, disengaged group."

Great start FJ keep up the communication, good and bad news, and the residents will help you as much as they can!

Dana

moreless

Looks like its gonna be a bumpy weekend for Residents!

Rolling Restart to deploy server version 1.24.5, Wed-Fri, Sep 10-12
We will be deploying server version 1.24.5 to Second Life this week. This version includes no new major features, but fixes bugs with the current 1.24.4 server that is on Second Life. More details of what’s been changed can be found on the Second Life Wiki’s release notes page for 1.24.
This rolling restart will follow our usual three-stage deploy:
Wed, 09/10, 9pm : A pilot group of ~3000 regions will be restarted.
Thu, 09/11, 5-9am : Half of Second Life will be restarted.
Fri, 09/12, 5-9am : The remainder of Second Life will be restarted.
As with all rolling restarts, each region will receive warnings starting 5 minutes before they are restarted. No region should stay down more than 10 or 20 minutes; if your region stays down longer than 20 minutes, please contact support.
Further discussion of the 1.24.5 server deploy can be found in this thread in the Second Life forums.

Server Crash Affecting Logins
[6:59 AM Pacific] One of our main infrastructure servers has suffered a crash. This will result in various in world problems as well as failed logins. Our technicians are working on damage control and we hope to have things back to normal and logins fully restored shortly.

Prospero Linden
The problems we had before (an hour or two ago) were not due to the rolling restart. A technician in our datacenter hooked up a cable, and somehow a network loop was created. We figured out what was going on pretty quickly, and asked them to disconnect the cable. In the mean time, however, there was a fair amount of chaos. We're not sure exactly how the cable was connected incorrectly...............
In case you didn't see it on the status blog -- we're postponing this morning's rolling restart, as we have one more bug fix that's going to go in. Assuming all goes well, we'll redo the pilot roll later this morning, and then will do a Thursday Night / Friday Morning half-and-half roll of the entire grid.

moreless

Warning GRID ISSUES

There is nothing on the blog (as usual) Nothing on grid status page either
but I have been having problems for a few days now with LAG levels being high, failures on teleport and rezzing objects.
Today attempts to rezz almost invariable fail - I would not attempt to buy anything right now either - teleport sometimes works and sometimes doesn't.
Right now I am stuck on the log in screen after 20 minutes and 3 attempts to log back in after trying everything else.
My guess is that maybe it has some connection to the 1.24 Server deploy but that is just a guess - timing seems to much to be a coincidence - come on guys I guess you got it wrong again!
Time 06.30am SLT
I will update you if we get to hear anything.
Dana

UPDATE:
Live chat help -
Aurelie: Hello dana_Vanmoer, what can I help you with?
dana_Vanmoer: I can't log in
dana_Vanmoer: just get stuck with the bar about a 1/4 way up
dana_Vanmoer: I have tried several times in the last 20 minutes
dana_Vanmoer: hello?
Aurelie: it is a known issue
dana_Vanmoer: so why isnt it on the grid status or blog?
dana_Vanmoer: and is there a solution or time frame?
Aurelie: they are trying to fix it
Aurelie: and they are going to put a note on the blog
dana_Vanmoer: so we dont need to know because they are trying to fix it?
Aurelie: not yet
dana_Vanmoer: so i just wait for the 'never there' blog updates?
dana_Vanmoer: a warning on the log in page may be a better alternative instead of people wasting their time
Aurelie: is there anything else i can help you with?
dana_Vanmoer: yes give me some time frame?
dana_Vanmoer: or you suggest i just keep trying?
Aurelie: keep trying
Aurelie: some accounts now manage to log in
dana_Vanmoer: ok thank you for your help
dana_Vanmoer: not much point logging in if nothing works

UPDATE:
[RESOLVED 6:44AM PDT] The problem has been fixed. Logins and teleports should return to normal once the initial rush to log back in subsides.
We are having a problem with a VPN that we are currently investigating. This will have an effect on teleporting as well as blocking some residents from being able to log in. We will have updates on the situation as soon as possible.
(we shall see)

moreless

SL Grid Trouble - Avatars in the Mist

I logged into Second Life at about 8:30 PM SL time on Monday evening. I appeared as a wisp as first, which was no surprise. What was a surprise was finding everything but me was rezzing. I was still a ghost. Seeing some friends were online, I IMed a couple. I got no response.
Then I got a group IM, saying Second Life was suffering "split servers" and some people would not be able to reach certain others in IMs, "Group IMS still work, though."
Meeting up with four friends, three where ghosted as well. The fourth, who had a male avatar on, had been ruthed and was feminized. One saw me not as a ghost, but as ruthed as well, "nice breasts Bix."
Looking at the Grid Status reports:

[8:45PM Pacific] Many residents have reported that their clothing are not loading and they are seeing others as white clouds. There are also some other issues such as some regions down, inventory not loading and search not working. We are aware of these problems and are working to fix these issues as soon as possible.[8:09PM Pacific] Please be aware that some residents are also experiencing issues with teleporting. We will update this blog as soon as we have more news.
[7:32PM Pacific] SL is experiencing some network difficulties. Some residents have reported problems with loggin in. Please avoid conducting inworld transactions and purchasing lindens for the time being. We will update this space with details as soon as we get them.

There was nothing left for us to do but to log off SL and hope things were better the next day.
Bixyl Shuftan

moreless

Eye on the BLOG, new log in page

OOOoooooo a new shiny interface!!!
'We have made a few changes to the Second Life login screen, as you may have noticed, and we hope you’ll find them useful. You will see headlines from two sources, the most recent posts from the Official Second Life Blog and also the most recent Second Life Grid Status Reports. You can also subscribe to either feed, directly from the login screen, using the handy new RSS buttons.'

Reading through the comments after this post is great fun - for anyone needing a laugh I highly recommend it - some points of interest that have been brought up are extremely accurate.
It is good to see the attempt at feedback for those that the polls work for, although to my mind it is very limited, sometimes the answer is not a simple yes or no so i must therefore decline to answer, which of course would not be recorded.
The questions don't seem to be that relevant to customer experience, but it is good to see some Lindens actually taking the time to find some kind of feedback since the reports page still has no comment option.

One of the main problems with any RSS is that it's often way behind what is actually occurring at that given moment and as Katt Linden has said:
'The folks who post to the Status Reports are, themselves, the people working on the problems — the team designates one individual who updates that report. Sometimes they don’t post details because they don’t yet know the cause of problems. Sometimes issues are not widespread, for instance whatever had some experiencing login failure just now, and in cases like that the Status Report might not always be updated. (I don’t know what happened there, I’m simply listing some reasons I can think of why the page may not be up to date in terms of your own experience of the grid.)'
This brings up a couple of points:
1, If multiple people are reporting on a problem there are probably double the amount who actually experience it and don't know how/want to report on it.
2, Even if the people working on problems don't know what is causing them or how long it may take to fix it residents still need to be warned and to know they are not the only ones facing the issue, be it transaction, log-in, database failure etc........
3, It should be updated as soon as an issue becomes a problem and also warning sent out inworld (maybe a link to the grid status inworld so people don't have to open a browser to check).

The surveys may work out to be a good feedback tool eventually once the questions are honed better towards what a Resident is actually looking for (not an easy task and not one I would want) Christa Linden had this to say about them:
'We’ve initiated additional surveys to sample Resident attitudes across a range of subjects. We are experimenting with this as a new channel for communication and expect to hone the questions over time. Currently there are about a dozen questions on rotation. Some issues currently addressed include: brand awareness (”Have you seen the inSL logo used in your community?”), brand comprehension (”Do you know what the Second Life Grid is?”), popularity trends (”What is your favorite Second Life mainland community at the moment?”), and learning tool effectiveness (”Have you viewed Second Life Tutorials? What skills have the Tutorials helped you learn or improve?” ) among others. We hope that the data from these questions will help us understand common perceptions about the experience of being active in Second Life as well as identify the effectiveness of certain tools and programs.
We are looking into ways to share findings with Residents as well as how to extend these surveys in languages other than English. It will take time to get there. But we, as always, greatly appreciate your interest and patience!'

I guess more info is always good however we can get it but I will say that if the grid is offline and status and grid reports don't say otherwise it can be very frustrating as we Residents all know too well, so until this problem is fixed and Lag between the grid going offline and anything showing up on Log in screen or status reports then, to me, it's pretty useless but does give us something pretty to look at as we attempt to log in (P.S loved the suggestion to put a game on the Login screen for us when we can't log in and are waiting for the grid status to be updated LOL)
Dana

moreless

Eye on the blog

Well good news and bad news - typical of LL as we have seen!

To start with the good news (to an extent):

Second Life Grid Status Reports

Are you following the Second Life Grid Status Reports? If your business, your event, or your time with friends may be impacted by the Grid Status, we want you to have all the information we can get you, as quickly and easily as possible.
We’ve been posting Status Updates mixed in with a variety of posts on other topics. We realize that many of you don’t want to have to hunt through the blog for needed Status information, so in an effort make it easier for you, we’re following the tech industry standard and moving that information to a Status page.
Starting today, we’ll be collecting all the Second Life Grid Status Reports onto that one page. Now, you can check for an update without having to scan through all the other blog posts.
You can also subscribe to updates on your mobile phone using Twitter, or with an RSS reader.
Yep, get your Second Life Grid Status Reports, wherever you go.


Now this sounds like a great idea, but lets look a little deeper;

For a start it means a cleaner blog for the Lindens to show off to corporate business no more long posts with angry Residents comments filling up the page.

Then, of course, it takes away the ability to comment on the issue posted:

Yesterday I DID get an inworld message saying there was a problem with some accounts associated with one of the inventory databases; great, actual warning inworld! problem was an hour later, after a bad crash, I could not log back in so off I toddled to the blog to see what was happening - nope not there - ok, off to the grid status page : [05.51 AM RESOLVED] Logins should be restored for all affected accounts. It may take some time for things to return to normal as the system catches up and everyone starts logging back in, but they are back online. -Chiyo

Resolved??? um no..........

This was then at 06.25 hmmmmmmm...............

So what can I do? Answer NOTHING, there is no way to communicate that no the issue is not resolved and to be perfectly honest I couldn't be bothered to go through support and get a load of pre-printed pages telling me my firewall was blocking SL (this is what happened last time) so I just gave up for the day!

So in conclusion having up to date easy to find grid status - great idea, one place to find out whats happening, unfortunately most people find out more from Resident comments on grid status than from the actual posts saying [RESOLVED] they usually aren't.
The comments have been removed also effectively striking the Residents dumb, there is no recourse and no feedback here so what APPEARS to be better communication and easier to find info is actually the opposite.
Dana

UPDATE:
"As for the harsh things some of you are expressing, I’m sorry you’re frustrated. We are indeed trying to get comments enabled on the Status page."
A single click of a button turns on comments Katt - I am disappointed

moreless

Yes you can MAYBE log in, but why bother?

Unless you were lucky enough to log in amongst your friends you cannot get to them, TP is borked and I won't be trying to buy anything (The system is currently unable to process your request. The request timed out.). IMs work so at least you can talk (if your friends list loads that is) textures don't load, notecards ditto, oh and don't try to get dressed either because its a waste of time, I did eventually manage to get hair, but shoes? not there even after 15 minutes. It took 20 minutes to change pants and prims still havent appeared.
Walking? well if you call sliding across the ground with your hands on hips walking then that is at least possible
Uploading, don't even go there LOL
building seems to be possible and it even managed to delete the prims i rezzed which is a slight improvement from reports we have heard.

To give them their due (ROFL) LL are at least admitting they can't fix it and are keeping people informed:

[07:30AM 04/05/08] It seems the problem is even harder to nail down than we earlier supposed. We need to close logins again to go back to diagnostics. -Lotte

Hopefully we will all be back sometime this week with the new Havok4 working and transactions going through everytime, oh no thats just a dream I had once upon a time...........

moreless

What a wonderful world we have chosen to live in????

Yesterday saw record numbers logging into SL with a concurrency of over 66,000 online. While Philip Rosedale gets ready to go before The Congressional Subcommittee on Telecommunications and the Internet and talk about "Applications and Avatars in a User-Generated Medium" we, actually using it, struggle with huge stability issues and stale transactions that have troubled the grid for most of the week.

The stability issues have plagued residents to such an extent that after a few hours at peak concurrency the actual users online dropped by more than half as most gave up in sheer frustration.

"Our database has crashed and caused in world services to go down. We expect it will be ten minutes minimum before we are up and running again."

This was the message on the blog followed by yet again "Resolved" which of course did not last long, Logins were problematic as were in world transactions, teleporting, maps, search, attaching objects and of course the support portal (just when you need it most).

Together with the new ToS and branding issues it is becoming obvious that its open for a competitor to step in, in my opinion, if there was another platform out there with similar capabilities most of SecondLife issues would soon clear up as people took a better option.

Yes of course people will still sign up and yes businesses will look at it as an option for their venture into the virtual world, but with all the bad press and Linden Lab doing everything they can to put residents off and now stopping the very people who have promoted them freely, what business will they have to sell in 5 years time? As one resident commented "When somebody steals a texture or build from another resident, LL tells us that it’s got nothing to do with them - but when we use THEIR graphics they suddenly start caring about copyright. Just keep looking after number one, LL - it’s what you do best."

The worst of this new ToS is the manner in which they have yet again failed to deal with the people it affects the most, 'Please check out Brand Center FAQ, if you’ve still got questions, please drop an email to tm-questions@lindenlab.com - note that the trademark team can’t guarantee that they’ll be able to respond personally to all the emails they receive. If your question is urgent, you should contact a trademark attorney.'

Anyone that is not an attorney, and even those that are practiced in trademark law, cannot give a straight answer as the new branding, FAQ and ToS together are confusing and complex even to the point of those legal eagles not being able to agree on what is now allowed and what is not.

I look forward to some clarification from a press release supposedly forthcoming from Linden Research Inc®

sources: SecondLife blog massively

UPDATE: Several In-World Services such as teleporting, map usage, search functionality, IM/chat and image upload failure are being experienced across Second Life. We are working as quickly as possible to resolve these issues. We will update you here as soon as further information is available. Thank you for your continued patience. (unsurprisingly I cannot log in right now getting DNS server errors, NOT mentioned on the list of problems)

moreless