About Managed Content

The managed content system, which openBVE once had, did not enjoy success at last. I personally think such a system is of great convenience, as you do not have to search through the entire web to find a route.
However, quite a few other content distribution systems, such as BaNaNaS of OpenTTD, is running successfully for quite a long time.

I wonder what brought the managed content system of openBVE down, and, as openBVE now already has a decent package manager now, will a package manager which features downloading and automatic dependency resolving, like apt or yum, fit the requirement of the community?

I am actually quite willing to try to develop such a system. Aside from server running costs, and potential copyright issues, I would like to know the  problems inside the concept, as the previous one already failed.

Thanks!

I have considered using npm. NPM is a quite nice package managing system, which allows really easy uploading, with few limitations (so far), and is completely free of charge for publishing contents. We just need to integrate it with the current package management system of openBVE.
I tried to upload a package onto https://www.npmjs.com/package/@obpkg/br-signals, and it turned out to be fairly easy.

Right now it is a zip archive inside a tarball. npm extracts the tarball and brings the zip archive to us, which can be fed into the package management system of openBVE. I can just write a program, maybe called "obpkg", to automate this unboxing process. You just run "obpkg install br-signals", and all is done!
Modifing the packaging system of openBVE to adapt to npm would be a more graceful solution, but doing it this way will work so far.

I guess npm would be quite appropriate. It features flexible search api and easy uploading.
https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md
And it seems that npm would be quite a long-running project in the near future...

The original npm client does have trouble uploading large tarball files, because it tries to encode the entire file into base64 string in memory.
But as the api is quite simple, writing a custom client is proven to do the job.

Hello.
I also think it would be nice to have an online package system, as you say.

I'm not familiar with the previous online package system, but there seems to have been some licensing issues.

Please take a look at the post on the site below.
https://forums.bvestation.com/index.php?topic=1361.0

Therefore, since the package was licensed only in the public domain, it is likely that there were authors who wanted to use the system but could not.

Now, about using NPM, I have some doubts about that.

Does NPM host files that have nothing to do with JavaScript, such as OpenBVE?

Does its usage meet NPM's terms and conditions?

Unless these two points are proven to be acceptable, it will be difficult to use NPM as it is.

I think it would be better if we hosted the package ourselves, even if we used the workings of that package system.

S520 wrote:Does NPM host files that have nothing to do with JavaScript, such as OpenBVE?
Does its usage meet NPM's terms and conditions?

Thank you!
You got the exact point. According to the policy of npm:

https://www.npmjs.com/policies/open-source-terms#acceptable-use wrote:Packages that are not functionally compatible with the npm command-line client. For example, a "package" cannot simply be a PNG or JPEG image, a movie file, or a text document uploaded directly to the registry. Using the Public Registry as a general purpose database is not allowed.

Of course writing some JS code, such as a self-installer, or maybe something that does something nice about this package with openBVE, would satisfy this statement. Just make a CLI node.js program which installs this package, and the entire package is thus considered a part of the resource of this program But it is just taking advantage of the loophole in the policy, and it all depends on the kindness of npm administrators to decide whether it is appropriate or not. However, As npm, Inc. is now being purchased by GitHub, which brings them rich financial support, and might hopefully make the community permissive to such packages.

I agree that managed content system should be more permissive about copyright issues, as it is really hard for a BVE route to be 100% CC compatible. Maybe one should just screw it and let authors manage the copyright on their own. When copyright violation occurs, just blame the pirates, remove the content on author's demand, and state the content system is only responsible for holding files.

The main problem of hooking up servers ourself is that they are expensive. Storage space is expensive, and high network flow is expensive. Setting up servers with residential network could be low-cost, but the overall performance would be poor. So it is quite a difficult issue.

To make things worse, I live in PRC, and the local authority is so damn good at blocking everything nice. That means OneDrive, MegaUp, Amazon S3, and etc. are not options, at least for me and my colleagues.

How about having each author provide storage for the files themselves, and our servers host the download links and dependencies and other information?

It may not be a very good idea, but it means we don't have to have huge storage.

S520 wrote:How about having each author provide storage for the files themselves, and our servers host the download links and dependencies and other information?
It may not be a very good idea, but it means we don't have to have huge storage.

Well, that means we are going to find some nice web drives, and encourage our authors to use them. That also introduces instability to our system, as some of them can shut down at some time. And the services have their limitations.
For the aforementioned reason, Amazon S3, Google Drive, OneDrive and MegaUp are not in my consideration.
Github Pages, or Gitee Pages are not quite good, since they have a limitation of 100MB for each file.
Github repository can be considered, but that requires the content of the zip archive being uploaded rather than the entire zip because of the 100MB individual file size limit. The hard limit is 100GB and soft limit is 1GB for each repository. But to make things worse, ISPs here are instructed by authority to deliberately drop around half of the tcp packets sent to github.com, which makes uploading stuffs to github extreeeeeeemely slow. Also, raw.githubusercontents.com is not available.
Gitee repository has a limitation of 500MB for each repository, which is not quite enough for high-quality routes.
TeraCloud seems a nice option. It features 5GB storage space for each person for free, direct link access, and without file size limitations. However, it is based on WebDAV protocol, which means reading and writing the files both needs a password. So the password must be published to let players download files from teracloud. Also, 100% tcp packet loss is reported by one of my collagues when he tries to connect.
I have seen uploader.jp being used by japanese route authors. The problem is that it does not feature direct link, and has to be accessed through a fully-functional browser because of the cloudflare DDOS protection.
Honestly speaking, NPM is the cloud storage with fewest limitations I have ever heard of. I would like to know if there is better candidate.

Talking about homemade server, you can see https://api.zbx1425.tk:8953 , where release and nightly builds of openBVE are mirrored. I refitted the router at my home to serve as a web server. Currently it is running on an 64GB USB stick, and purchasing a portable hard drive could get me around 1TB of storage space. The major issues are the unavailable port 80 and 443, and the download speed and high-flow performance might not be quite nice.

As I recall, the original "managed content" proposals differed very substantially from the current "package management" methods. At the time I looked at the system devised (I believe) by Michelle and though I would have liked to avail of it, to be perfectly honest and after reading the guidelines, I couldn't understand a word of it. And I know I wasn't the only one who couldn't work it out.

There seemed to be a gulf between the minds of the program makers (who all understood programming languages) and the "grunts" doing the spade work toiling with the simple instructions to make the routes etc. I think I'm qualified and entitled to say this, having made a LOT of routes and rolling stock, I don't understand things like coding and Linux systems. To me, C# is something I play on a guitar! The simple route and object code given to me by the programmers is fine, that and native Photoshop skills lets me build stuff.

Now, once Chris had contrived the package management facility we now have, that was great. I could do that within my limited technical ability. So I adopted it straight away for all my stuff right from the go.

So, you see, it needs to be kept simple. There's few enough route creators in UK / Ireland as it  is, without scaring any potential candidates away. Anything more complex than what we have now and you can kiss goodbye to my stuff, because I won't be signing up to it.

graymac wrote:So, you see, it needs to be kept simple.

After several days of discussion with Mr.S520, we have decided the basic outline of this system, I set up a database, and Mr.S520 set up an application on heroku to test it out.
This system will be entirely built upon Mr.leezer3's package system, it just provides a way to download packages from a central source, so the players will not have to struggle searching for the routes they wanted.
As storing files in a server costs a considerable amount of money. We decided to just store index of the files at the server, and use third-party storage services which provides convenient api to fetch files, such as Google Drive or Teracloud, to hold the files.
We have created a demo page to register packages (The finished one would be much, much more compact than this!), you can see it right here: https://obpkg.zbx1425.tk/dbedit/
And the file can be fetched by accessing https://source-server-test.herokuapp.com/file.php?guid={Insert Package GUID here} .
Of course the downloading would be all automatic when we finished coding it. So no worries! I think the best result is another tab pops up at the route selection panel, and when clicked presents a list of routes to you. Click on one and it would start downloading and installing it automatically.

So our system would work like this:
1. Create a package using the current package manager inside openBVE. Take a note of the GUID given by the manager when creating.
2. Upload the package to Google Drive and get a share link.
3. Go to our registry, punch the GUID, Name and Google Drive Share Link in.
4. Your package is online!
Of course we might need you to register an account before uploading them, to prevent some kind of spamming, or frauding, or something else like that. I and S520 agreed that we would not put any restrictions on the packages to be uploaded. Anything legal and not violating someone else's copyright would be fine.

The biggest issue with Michelle's system was always the licence condition.

That got the community's backs up, and it was from then on doomed from the start.
If it had had more sensible conditions attached, I'm sure someone would have run up a tool pretty quickly.

leezer3 wrote:If it had had more sensible conditions attached, I'm sure someone would have run up a tool pretty quickly.

That would definitely be considered. I think we can accept all original or authorized reposting, and let the uploaders handle the licensing problems themselves. And we remove the associated packages when abuse is reported. 
By the way, do you think integrating such functionality into openBVE is a good idea, or we should just create a standalone program for this?

The moment all this is too integrated into one thing, the ice under the "I'm just showing pointers, the uploaders are solely responsible" argument becomes legally thin. And by"thin" I mean "virtually nonexistent".

Quork wrote:The moment all this is too integrated into one thing, the ice under the "I'm just showing pointers, the uploaders are solely responsible" argument becomes legally thin. And by"thin" I mean "virtually nonexistent".

I see your point. The "uploaders solely responsible" pattern is an easy-to-implement way to absolve oneself from responsibilities, and neglecting the management would lead into chaos. But I am also afraid being too "responsible" may lead us into the Michelle's way, which essentially make most developers unaccessable to the system.
I personally think, if such a centralized system exists, it would neither directly encourage piracy nor discourage piracy. (Piracy there refer to all kinds of abuse like unauthorized upload, repost, etc.) And if it does, it only turns distributed piracy into centralized piracy. Which might be better, since as long as we have control over the "pointer table", we have easier control upon the centralized piracy, rather than the distributed piracy which we cannot control.

It is quite hard to balance. While we try to make things more difficult for pirates, it seems that we also unavoidably make things more difficult for developers. And as Graymac stated, if we made things too difficult, then developers would just turn away, and render the system useless.
I have not a single bit of experience on managing any community. So I think this part should be left to someone leading, and well-aware of the current situation in our community, to establish the rules. I would like to ask if Sir Quork have any idea on this.
And as the basic mechanisms of the system are being determined, I found this be like some form of "pointer table" as you have described, as we are leaving the storage part to third-party services like Google Drive to save cost. It seems like we are building a pointer table with direct link downloading and automatic dependency resolving abilities. Downloading routes with a single mouse click is certainly convenient, and it would also make developers publish routes easier by having a centralized index. I personally am a bit doubtful if it would worth the effort though.

Before you go down the road of producing a depository of material, here's a question I'd ask you to consider  -  and other makers of content might ask too.
The question is: "Why would  I want to go to additional time and trouble to put my content on a third-party site when I have my own website, which guarantees me control over the distribution and presentation of the entire range of Celtictrainsim material?"

As for helping persons to find route and rolling stock material, it's simple enough to promote it on this, or one of the other, user forums. For anyone who hadn't heard of my work there's a link to it (and that of others) on this forum page. https://bveworldwide.forumotion.com/?pid=3

The BVEWorldwide staff are happy, I know, to give a link to the websites belonging to producers of suitable material.

One further point, and this is my own policy / opinion, I can't speak for others. I would not be supportive of any system that would attemt to link DIRECTLY to any of my downloads  WITHOUT going through the normal route via my website. I consider that to be disrespectful, in effect "grabbing the candy while dissing the shopkeeper". So I only permit links to my site homepage, I hope that's clear.
Do call and see http://celtictrainsim.com/ 

That begs a slightly  different point-

Would this be better as a well curated online database that simply links to the authors homepage?

Most of the dependency etc. could be handled here, it wouldn't be quite as neat though......

Either way, I'm ambivalent as to this.
The aims are worthy, but whether the logistics work is another thing

graymac wrote:The question is: "Why would  I want to go to additional time and trouble to put my content on a third-party site when I have my own website, which guarantees me control over the distribution and presentation of the entire range of Celtictrainsim material?"
As for helping persons to find route and rolling stock material, it's simple enough to promote it on this, or one of the other, user forums. For anyone who hadn't heard of my work there's a link to it (and that of others) on this forum page. https://bveworldwide.forumotion.com/?pid=3

This is also what I have been thinking of, and that's why I opened a topic there and start discussing, instead of just put my head down and start punching 

The main problem of a personal website is that some people just don't know your website. It is more problematic in a non-English-speaking country like the one I am in. And to make things worse we do not have Google, and the local alternative Baidu is not doing a great job on collecting English websites.

I did a quick test on a few of my friends. They are skillful openBVE developers and have been playing for years, and have decent English knowledge. I can say that they are above the average level of the other players here.
I asked them if they knew celtictrainsim.com, and they said "Not yet"
I asked them if they knew the "websites worth a look", and they went like "What? BVEWorldwide even has a link section?"
Talking about the bve5 part BTW, as the main community is in Japanese, which non-speakers cannot even type, it took me 3 years to find http://wikiwiki.jp/bvets/.

The problem is that players don't know the websites. The "website + link" solution should theoretically work fine... except it doesn't here.
Quite a few local developers have set up their own websites for years, but still a lot of our players play our routes without knowing our websites.
For example, if I was a new player and have no idea of what openBVE is and where the community is, when someone recommended openbve-project.net to me, I would just click "download" and close the tab. To reach the bveworldwide portal page, you have to click "Links" -> "Further Links" -> "BVE Worldwide", and I cannot see what would motivate a new player to do this, especially when he or she does not know English well. Not to speak of as vps.bvecornwall.co.uk is of poor connection speed here, most players get openBVE by sending the installer exe to each other, and quite a few of them are actually PLAYING openBVE WITHOUT EVEN KNOWING openbve-project.net , which is quite sad.
I tried to set up a "mirror" website at https://api.zbx1425.tk:8953 to help, but it seems that it is also not well-known too.

And as the players don't know the websites of the developers, the secondary-distribution is greatly encouraged. Some players are happy to get route packages from secondary-distributors, because they don't want to spend a lot of time searching for the website and then struggling through all the English texts. For players they get the packages in an easier way than the official one, but for we developers this is clearly not what we wanted to see, as secondary-distributors often do not give credit to the original developers.
The worse part is people are making use of this imbalance in information. There have been several man who tried to sell the routes made by us. The players pay them because they don't know where else they can download them, our hardworks were made use of, the pirates get quite a few money without doing anything, and we, the original developers of the routes, were not credited, respected, and not a single penny got into our pocket.

So I personally think, if we are building an index:
1. It must be official, or half-official, like BVEWorldwide. It is hard for fanmade indices to be well-known outside the local fan community.
2. It must be placed on an appearant spot, so anyone who holds openBVE can know about it. I think we can place a "Download more routes" link in the main program, maybe as a extra item on the top in the route file list besides "..", or other appearant places.

But the second point is actually hard to achieve, since like I have said many players download openBVE by sending an exe file to each other on SMS softwares, which means they hardly updates, and it would be hard to let them be aware of the newer version. And some communities such as HKRSC have been deliberately suggesting their players to use openBVE 1.7.0.0, as they consider the versions after that "not stable enough".

The shopkeeper point is actually easier to achieve. We can add homepage as an item in the registry, and just call the system browser and let your homepage pop up (well, we should add a command-line argument to disable the behavior) every time the player downloads a route.

leezer3 wrote:Would this be better as a well curated online database that simply links to the authors homepage?
Either way, I'm ambivalent as to this.
The aims are worthy, but whether the logistics work is another thing

I am also ambivalent about this.
Once I and S520 decided to use third-party services like Google Drive, the logistics part is actually essentially same, as we are only holding links instead of holding the files. It is only the problem of which links to hold, the links to homepages or the links to Google Drive which directly gives you the file.
As quite a few contents can be reused, I think it would be nice to have some kinds of automatic dependency-resolving.
Because the primary reason why I would like to want this is to counteract secondary-distributors, I actually hope the system can make downloading easier. Players turn to secondary-distributors because downloading from them is easy, and finding original website is hard. TLDR it is a system for lazy men. Giving the credits to the original author is easy, as we can show some kind of license of other info before downloading, such as:

BR Signals Open
Authors: Anthony Bowden, Steve Green, Tony Haylor and Simon Gathercole
BRSigs_Open is released as public domain material.

I have played OpenTTD, and its content system "BaNaNa" really astonished me. It is what I hope openBVE can also have. If there is such a convenient way to download packages, who on earth would even turn to secondary distributors!

Let me add a few more from me.

leezer3 wrote:Would this be better as a well curated online database that simply links to the authors homepage?
Either way, I'm ambivalent as to this.
The aims are worthy, but whether the logistics work is another thing  

Unlike a service like Wiki, the server we are thinking of works like a reverse proxy.
So, if our servers are accessible from China, users in China can download add-ons that are stored on Google Drive, etc. that are not directly accessible from China.

Also, what we are suggesting is just one of the distribution methods.
The current distribution method is kept intact.

S520 wrote:So, if our servers are accessible from China, users in China can download add-ons that are stored on Google Drive, etc. that are not directly accessible from China.

Also, what we are suggesting is just one of the distribution methods.
The current distribution method is kept intact.

I personally think the overall aim of this service is more than finding a way to bypass the network censorship though. The censorship is actually the easier part there. The main goal is to provide a easier place for the players to find and download a route.

As most of our players (at least here in China) are not technology freaks and super scholars, it seems that they are just playing the game for fun, and are not willing to dig around the internet to find routes to play. They simply ask for help instead, and such posts flooded local forums.
Of course we can neglect this, which would hopefully eliminate the lazy players and make the community into an elite community - But unfortunately it didn't happen. While there is demand, there is market, and secondary distributors began popping out. Some of them are just people who have a copy and wanted to help when asked, and some of them are jerks who violate the copyrights to satisfy their vanity. But the thing in common is that they often neglected to mention the original author, which took away the respect the authors deserved.
So now it became a community full of newbies and secondary distributors. The appearance of these lazy newbies seems preventable. As long as an community is expanding, there would be some people who are not quite "into" it appearing. And you cannot simply blame everything on them though, as we have been them once a while.

So I thought about creating a way to download packages right inside the game, just like OpenTTD, to compensate these lazy dudes. As players can get packages from an official method in a very easy way, secondary distribution would be inhibited.

Also, it would be more easy for developers to get their routes well-known by having an globally available, official index.
If you publish your route on your own website, only your followers would know that.
If you publish your route on BVE Worldwide, only some English-speaking people would know that.
If I publish my route at Hmmsim group at Baidu Tieba (Where most BVE players in China gathered), I'm sure you would not know that.
However, if we have an globally available package index integrated officially into openBVE, everyone on earth playing openBVE would (hopefully) know that when one publish a package.

By the way, I tried to write a documentation on the planning of the system.
https://obpkg.zbx1425.tk/apidoc.html

leezer3 wrote:Would this be better as a well curated online database that simply links to the authors homepage?
Most of the dependency etc. could be handled here, it wouldn't be quite as neat though......

The first point is that popularity cannot be guaranteed unless we build the link directly into our program at an appearant place so that all players can get to know this.
The second point is that dependency resolving and downloading would be more difficult as it requires manual interaction, and language can also be a problem.
The third point is that if we have a database for packages instead of websites, we and the players will have a better view on the motion of our community. For example, if a route gets released, a database for packages sorted by date descending will make players around the world aware of it immediately.
The fourth point is that websites, especially some of the free hosting services, are in high risk of shutting down. Quite a few resources have been unavailable due to the authors' websites being shut down. As Mr.S520 suggested, we can use Google Drive to store the files directly, and it is likely to be running in long-term.
The last point is network auditing in China, which Mr.S520 has discussed before.

leezer3 wrote:Either way, I'm ambivalent as to this.
The aims are worthy, but whether the logistics work is another thing 

Mr.S520 have managed to set up a server entirely on the free-of-charge Heroku platform. And I wrote some scripts to get the most basic architecture running. Api documentation can be found at https://obpkg.zbx1425.tk/apidoc.html .
I have started to write a CLI client at https://github.com/BVEContentService/OBPkg , but little has been done so far.
As Mr.S520 is out of free time now, and I am not familiar with the package handling inside OpenBVEApi.dll, I really would like to hear a word from Mr.leezer3 about this.