Guild icon
Direct Messages
Private / Riot
Avatar
SalC1 29-Jan-23 11:48 PM
Hey Riot, I'm currently working on a YouTube video about your server, Minecraft Online, regarding the current state of the server. So I thought I should talk to the man himself to get some direct answers to some questions I have about it. I understand you currently have a life outside the server, but if you have any spare time, I'd love to arrange an interview with you. I'm not sure if you use Discord very often so I'll be contacting you via other platforms in case this message doesn't get through.
Avatar
Riot 30-Jan-23 01:04 PM
hey there. I don't really use discord, but I try to catch up occasionally. Happy to try to answer any questions you might have - fire away
Avatar
SalC1 30-Jan-23 03:37 PM
Thank you so much for responding!!! You can answer these questions any way you’d like. The reason I’m asking you all this is because Minecraft Online is extremely important to me and many others in the Minecraft community mainly due to its extremely historic value. So I wanted to make a video showing the importance of this server and some of the long time issues it has had that had made playing on the server in practice, not very appealing. I understand you haven’t been very active recently with the server itself, which is understandable as some can only be interested in block game for so long. So I’m not sure how much you’d know the answers to the current issues the server has. However, due to the inconsistency of answers I’ve been getting from staff and players, I thought I needed to contact the man himself to hopefully get some solid answers. Most questions are my own. Others were asked by former staff/regular players. These will be marked with an *. 1. I have to of course ask first, why have you been (at least perceived to be) so distant from MCO these past few years? Do you have any future plans of ever returning? 2. One of the largest issues the server has faced for the past few years is the lag. I’ve received a variety of answers from Admins regarding why the server performance has been so poor. a. Some examples of what I was told: i. “Java memory leaks” ii. “if we knew what caused the lag it would likely be fixed” iii. “the timing out players thing seems to be more of an issue in 1.12” iv. Old server hardware v. Plugins like craftbook vi. Mega farms b. So my question is why haven’t these issues been fixed? What is stopping the Admin team? 3. Based on the last question, I heard a lot that the reasoning not much is being done is because of a lack of staff or lack of time to make these fixes from staff. Is this true? If so, would it be possible for others who have the time and know-how to help fix these performance issues? 4. Why has the server historically been stuck on older versions of Minecraft? 5. *Why do you keep inactive staff on the team? 6. *Why has the server been using obscure server software like Sponge rather than something more common like Spigot or Paper? 7. Last year, I made a $1,500 donation to the server as I was told that it would pay for main server hardware upgrades. Only recently have a received a message from an admin saying that that all the money was instead used for a completely different server, the test server to my understanding. Apparently, “dev availability meant that we were never able to move the server over.” What makes moving a server to new hardware so difficult? Why was this donation seemingly wasted on a test server? I apologize about the questions being mostly negative in connotation, but the community has just been extremely frustrated and wants answers. We care about this server immensely due to its historic nature. I hope that by getting answers to these questions, we can create and suggest solutions to the issues that the server has so that it can succeed for the years to come.
Avatar
Riot 03-Feb-23 11:22 AM
hey, apologies for leaving you hanging for a few days. I've recently had a bereavement and things have been quite busy.
11:22
As you are aware, I take a mostly hands-off approach to running MinecraftOnline, entrusting almost all the essential functionality to our very capable admin volunteer team. I'm not always up to date on smaller matters around the day to day running of the server, but I do keep abreast of more serious issues, and try to offer advice and guidance when it would be appropriate and helpful. As a result, I may not have very specific answers to each of your questions, but I'll do my best, and I can point you to team members who can help further in other cases.
11:22
I'll answer number 7 first, because I have recently caught up with the latest aspect of that situation, and I think it's an important one. I recall processing your donation manually at the time, as we were having some issues with paypal. The admin team selected a particular server available with our preferred hosting provider, Hetzner, and we calculated a hosting budget for approximately 10 months of uptime for the new server (excluding existing services), that we estimated this donation could fund. It was anticipated that there'd be up to a couple of months of overlap, where we would be running the new server in parallel with the existing production server, required for a smooth migration.
11:22
I placed an order with Hetzner for the server as selected by the admin team, created access credentials for the admin team, and made an announcement about the new machine, when it went live on 11 April 2022. That was where I left my engagement with that issue, with the process of migration delegated to the admin team, comfortable with the knowledge they had accomplished similar migrations smoothly in the past.
11:23
Image attachment
11:23
More recently, I noted that we were approaching the 10 month mark, and requested an update on the situation from the admin team. I was disappointed to hear that due to limited human resources and too many parallel projects, it had not been possible to complete the migration in the time allocated, and the service was still running on the old host. We had been paying for both old and new server in parallel. After review of the situation, and getting a prognosis from the admin team, it was clear that it was not a good use of funds to continue paying for the higher spec server when we were still not ready to migrate, and I made the decision to decommission the new server before the February billing cycle. The current plan is to commission a higher-spec server once the admin team has the time available to migrate in an efficient way.
11:23
Speaking frankly, I was rather disappointed with our handling of the situation. I feel we completely dropped the ball on this one, and find it's very regrettable that we failed to put your generous donation to good use. However, our admin team had the best intentions at all times, and knowing how busy they are - as volunteers, balancing real life as well as their MinecraftOnline contributions - I continue to be very impressed by their achievements over the last 10 month period. It's just a shame that a server upgrade did not end up being one of them. In retrospect, we should have waited to place the order for the new server until we were ready to coordinate a migration with minimal overlap and waste. Some important lessons have been learned as a result of this, and our procedures for planning future upgrades are certainly going to be more efficient.
11:23
I'll try to address your other questions next.
11:23
>1. I have to of course ask first, why have you been (at least perceived to be) so distant from MCO these past few years? Do you have any future plans of ever returning? It's a good pair of questions, and the answer is simply that there's never as much time as I'd like to have. I founded MinecraftOnline purely as a hobby activity, a way to blow off steam, 11 years ago. A lot of things have changed in the time since then. After its first year, it began to grow significantly, and we started to accept donation funding, and invest in more and more significant server resources. Over the following year our in-house development exploded exponentially, and we went from an almost off-the-shelf modded Minecraft server, to a completely unique offering - if I recall correctly, we clocked past a million lines of original code before the end of the second year of operation. Of course, as it's been said, measuring software in terms of lines of code is like measuring aircraft in terms of weight - it gives you an idea of how huge and unwieldy something is, but it doesn't exactly correspond to quality or performance.
11:23
As player demand grew, so did our content and software - and as a result, so did the ongoing effort required to improve and maintain it. Our moderator, admin and developer teams necessarily increased in size. Despite this, the workload per staff member continued to increase. Development focus shifted from expansion and improvement, to polishing and paying down technical debt, and developing in-house tools to make administration easier. Through this time I focused on reducing the "bus factor" - at the start, I had done everything myself, written every script, tweaked every configuration, set up every service. It took years to transfer this knowledge to our incredibly capable admins, but they are now in a position to do absolutely everything I was once needed for - and almost all cases, they can now do it better than I could, if I began to take a hands-on approach again.
11:23
This is just as well, because the amount of free time I have had available to spend on MinecraftOnline has also reduced tremendously. But I never really "went away", I just stepped back from having an active role in the public MCO community, limiting the majority of my interaction to discussion with the admin team only. I realise this makes me a lot less visible to the community at large, and has turned me into a bit of a mythical figure for many. That also, in a way, makes it more difficult (or at least, very taxing) to spend any amount of time in active chat, or in the game itself. I simply get mobbed by dozens of players, and as well-meaning as they are, I can never reply to every question I get - it is very rare I can find the time to write something like this in response to a query, and I suppose that leaves a lot of people feeling ignored. The truth is I hear a great deal about what goes on in the community, and I do offer my input to the admin team frequently, but I am very happy to let them handle the majority of situations that arise - I feel that after years of experience working together, they are now very much in tune with my original vision for the server, and are very much able to do the right thing without any need for hand-holding from me. In fact each of the admins are very capable mentors themselves, both to each other and to the moderator team. I feel confident taking a hands-off approach primarily because of my supreme confidence in their capabilities. I can only hope the intense sense of pride I have in them, comes across clearly, in the way I am explaining all of this to you.
11:24
I understand that your original question was more or less personal in nature, and I answered it by talking about the team and the server, so I'll try to remedy that a bit. I suppose part of what you're asking is not necessarily why I'm not so active, but where I've actually been. I hope you can understand why I choose to be quite private, considering the intense scrutiny that this sort of position attracts, but I'm happy to give an overview from a more public perspective. A couple of years after starting MCO, in 2012, I launched VoxelStorm - an small indie game studio - and focused my time almost exclusively on that until the release of our fourth game, sphereFACE, in 2017. After that, I put VoxelStorm on hold for a while, as my career moved more towards finance in London, where I spent a couple of years as Executive Director at Goldman Sachs - a role that allowed me very little free time, and frowned on outside projects due to compliance concerns. What little free time I had at the weekends was consumed by my directorship of British Mensa. After my resignation from that role in 2021, I have taken a slightly more active role in MCO once more, catching up on more social issues and being more actively engaged with administration - although I continue to be very busy, occasionally working in the crypto space, and having recently launched a new game studio, Armchair Software.
11:24
You asked if I plan to return. I don't really consider myself to have ever really gone away - I don't plan to start playing Minecraft on a daily basis again any time soon, but I also intend to remain engaged with the running of the server, offering guidance to the admin team whenever they need it, and indeed trying to answer questions from the community on the very rare occasions that I'm needed to do so.
11:24
>2. [Regarding lag] I know it must be frustrating, but there is never going to be a single simple answer. Every answer you've received is certainly a factor, and even those together are probably not the whole story. Some latency issues arise at the client side, some arise in networking, most server-side. Key processes of the minecraft server are not adequately multithreaded, and some seemingly light tasks trigger significant memory allocations and subsequent garbage collections. Because of deferred garbage collection, actual latency may not correspond directly to its cause. The operation of many in-game features contribute to this, including mob AI (hence impact from farms), Craftbook, and many smaller components interacting together/ I'm answering in abstractions here because you will get a much clearer and more up to date set of technical explanations from the admin team. The admins do now have profiling in place, and there is ongoing work to address the known causes of latency, but my understanding is that new issues are discovered periodically. Ultimately the server and its plugins are fairly feature-rich, and in the past we've focused on expansion rather than optimisation. Now, I understand the focus is on consolidating what we have and making small improvements where possible, including to performance. This may appear like stagnation from an external perspective, but it is essential for the long-term operation of the server.
11:24
>4. Based on the last question, I heard a lot that the reasoning not much is being done is because of a lack of staff or lack of time to make these fixes from staff. Is this true? The admin team's bandwidth is indeed limited, but it's not at all true that not much is being done. The admin team are working on a daily basis on improvements - it's just that rather than new features, most of it is beneath the surface. I'll quote Evan in his description of some of the recent developments: "...The data migration and website changes, which are important even if they are not as flashy as some brand new block types. One issue with being such an old and complex project is "technical debt," where solutions from the past create problems for the future. The idea that Minecraft would continue to be the dominating game that it is twelve years after its launch was not a given, which meant that in the early days, short term progress (in combination with Mojang's ever-changing plans) would create problems for future developers to solve. Many of our core server systems were written at a time before name changes, and a lack of developer availability in the mid-2010s meant that our systems were never able to adapt to the migration from player names to UUIDs. This is the reason that your /friends list might have players who don't exist and never shows the current names of players. It's important that we fix those issues, especially as our playerbase has grown following the update to 1.12.2, but those fixes take time, and there is only so much time in the day."
11:24
... If so, would it be possible for others who have the time and know-how to help fix these performance issues?
Absolutely. We've always been happy to work with any volunteer developers who want to offer their time. It's important to realise that working with volunteer developers outside of the core admin team also has its cost. Every contribution has to be extensively reviewed and tested. While the majority of our contributors are both capable and benevolent, there have been exceptions (including an obvious recent set of events), so I'm sure you can understand why it's not always a top priority to onboard new unproven volunteers as software contributors. It can also challenging to assist in matters such as profiling, or attempting performance improvements - necessarily quite intrusive - without being able to guarantee a chain of trust. But if your point is to make a call to the community to come forward and contribute their time, then please go ahead - our arms are open wide, especially to developers who do not need a great deal of hand-holding to get started.
11:24
4. Why has the server historically been stuck on older versions of Minecraft? 6. Why has the server been using obscure server software like Sponge rather than something more common like Spigot or Paper?
I'll let Evan answer these two together:
11:25
1. The development experience — we develop most of our plugins in-house and we wanted a platform that could support our ambition. Plugins like PenguinDungeons, which powers the Shulker Ranch and Dragon Fights, is a testament to both the ambition of our developers and the versatility and power of the Sponge platform. The Bukkit API (which powers Spigot, Paper, and other more well-known platforms) is old and inflexible, running effectively in maintenance mode. There are parts of the Spigot API which don't even support basic newer features like Text Components (introduced in 1.7.2! That's before the old 1.7.10 that MinecraftOnline used for years!) and cause general developer frustration, even if they can be more performant.
11:25
2. Copyright concerns — Server platforms which implement the Bukkit API (Spigot, Paper, etc.) do so on top of a literal patchwork of changes to the original CraftBukkit, a project marred with DMCA-related issues stemming from Mojang's purchase of the project. At the time that we were deciding on a server mod to move to after Canary, a lot of this was still unfolding (we updated to 1.7.10 during this process knowing that we needed to find a new mod afterwards) and the potential legal and support implications led to us planning to move first to Forge and then to Sponge. [Regarding staff bandwidth], much of that stems from the concerns I wrote above about Bukkit, et. al., as well as the lack of developer bandwidth. That is why it took so long to leave 1.7.10. As for the amount of time it has taken to move from 1.12.2 to 1.19.3, besides the obvious moving target, is that 1.13 brought massive changes to how Minecraft runs. This includes a completely new command system, "the flattening" (where item and block IDs completely changed internally), and large changes in order to allow more of the game to be driven by datapacks. Other servers cope with this by using these older platforms that ignore these changes (or implement them in not-ideal ways) but we want to get these changes right.
11:25
5. Why do you keep inactive staff on the team?
With moderators, we generally don't. Early on in the history of the server, I established a set of performance metrics for moderators, which the admins use to appraise moderators' performance and offer them feedback. Activity metrics are a big part of that, and as a result, most of our moderator team remains fairly active, as far as I am aware.
11:25
As for admins, we have always had a different approach - some of our founding admins have been inactive for many years, but periodically return and offer excellent contributions when they do. Because the admin team tends to be more technical in nature, there is a bit of a split here - I suppose you can look at them as "technical admins" or "community admins", the former focusing more on technical administration and development leadership, and the latter focusing more on social and community development, moderator leadership. The latter tend to resign if they are likely to be inactive for any period of time, but the former tend to stay on our books indefinitely - and when they come back, sometimes years later, their contributions are usually of very high quality just as they were before they left.
11:25
Because of the inherent seniority implicit in the admin role, our admins also tend to be at a different stage of life than the majority of our (younger) playerbase and many of our mod team, and often the amount of free time they have available can fluctuate significantly from year to year. Thinking of this, and out of respect for their hard-earned admin role (which usually takes many years of active presence to reach), we do not usually consider inactive admins for demotion.
11:25
We don't have any concept of "slots" or "roles" for individual staff members - we are always keeping our eyes open for the right talent to promote, and I have been personally involved in every admin promotion in the history of this server. There's no need to worry that inactive staff members are somehow preventing new talent from being promoted - when it is clear someone is appropriate for the role, we begin consideration of promotion - and this process can take a long time. Over the years we've never regretted waiting to promote someone, although we have regretted failing to wait. Like a fine wine, our admins have all matured a long time in the bottle!
11:26
I understand that from an external perspective, it can sometimes feel like faithful moderators are being passed up for promotion, or that there are individuals who are keen to contribute in some way, but are not given the access they would need to do so. I hope that recent events have served as a good demonstration as to why we are generally so cautious about promotion, especially to admin level, and why we can never be too careful with regards to the access we give to technical contributors, no matter how well-meaning they may appear. The admin team are always grateful for recommendations and feedback from the player base, and this includes recommendations for promotion or other forms of recognition for members of the community they wish to vouch for. Ultimately, if anyone wants to come and help - please come forward, and the admins will try to put your talent to good use wherever possible.
11:27
I know there's a lot of text here - let me know if there's anything I've missed, or any more detail you'd like me to fill in. You're also very welcome to approach the admin team directly as I'm sure they'll be very happy to answer your technical questions in much more detail. (edited)
Avatar
SalC1 03-Feb-23 10:11 PM
Thank you for taking your time to respond to the questions. I hope that this will bring some light to the frustrations some of the players on the server have had. I also wanted to see if it's alright if I publish your full, unedited, response alongside my video? As I'm sure people will want to see it.
Avatar
Riot 04-Feb-23 06:26 AM
Sure, feel free to share any or all of my response as appropriate. I would also be interested to hear a summary of the frustrations you've heard expressed - even if it's things we're already familiar with, detailed feedback is always helpful
Exported 30 message(s)