Triplehx3/serverlist
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: Triplehx3:rework
Branches Tags
Triplehx3:master Triplehx3:rework Triplehx3:boost-domains Triplehx3:mongodb
..
compare: Triplehx3:mongodb
Branches Tags
Triplehx3:master Triplehx3:rework Triplehx3:boost-domains Triplehx3:mongodb

1 Commits
rework ... mongodb

Author SHA1 Message Date
ShadowNinja
46ff8cdaf0 Add support for persistent storage using MongoDB 2016-01-31 14:17:06 -05:00
30 changed files with 698 additions and 1579 deletions
Expand all files Collapse all files

7
.editorconfig
View File

@@ -1,7 +0,0 @@
root = true
[*]
indent_style = tab
indent_size = 4
trim_trailing_whitespace = true
insert_final_newline = true

16
.gitignore vendored
View File

@@ -1,11 +1,7 @@
*~
*.mmdb
*.sqlite
node_modules/
__pycache__/
/server_list/static/list.json
/server_list/static/servers.js
/config.py
/celerybeat-schedule
/package-lock.json
/Pipfile.lock
node_modules
__pycache__
static/list.json
static/servers.js
config.py

18
Pipfile
View File

@@ -1,18 +0,0 @@
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"
[packages]
maxminddb = ">=2.0.0"
Flask = "~=2.0"
flask-sqlalchemy = "~=3.0"
flask-migrate = "~=4.0"
celery = "~=5.0"
[dev-packages]
pylint = "*"
rope = "*"
[requires]
python_version = "3"

241
README.md
View File

@@ -1,204 +1,109 @@
Minetest Server List
Minetest server list
====================
Webpage Setup
---
Setting up the webpage
----------------------
You will have to install node.js, doT.js and their dependencies to compile the server list webpage template.
You will have to install node.js, doT.js and their dependencies to compile
the server list webpage template.
First install node.js, e.g.:
```sh
sudo pacman -S nodejs
# OR:
sudo apt-get install nodejs
```
# apt-get install nodejs
# # OR:
# pacman -S nodejs
# # OR:
# emerge nodejs
Then install doT.js and its dependencies:
```sh
npm install
```
$ cd ~
$ npm install dot commander mkdirp
And finally compile the template:
```sh
cd server_list/static
../../node_modules/dot/bin/dot-packer -s .
```
$ cd static
$ ~/node_modules/dot/bin/dot-packer -s . -d .
You can now serve the webpage by copying the files in `server_list/static/` to your web root, or by [starting the server list](#server-setup).
You can now serve the webpage by copying the files in static/ to your web root, or by [starting the master server](#setting-up-the-server).
Embedding in a Webpage
---
```html
<head>
...
<script>
var master = {
root: 'https://servers.minetest.net/',
limit: 10,
clients_min: 1,
no_flags: 1,
no_ping: 1,
no_uptime: 1
};
</script>
...
</head>
<body>
...
<div id="server_list"></div>
...
</body>
<script src="list.js"></script>
```
Embedding the server list in a page
-----------------------------------
Server Setup
---
<head>
...
<script>
var master = {
root: 'http://servers.minetest.net/',
limit: 10,
clients_min: 1,
no_flags: 1,
no_ping: 1,
no_uptime: 1
};
</script>
...
</head>
<body>
...
<div id="server_list"></div>
...
</body>
<script src="list.js"></script>
1. Install Python 3 and Pipenv:
```sh
sudo pacman -S python python-pipenv
# OR:
sudo apt-get install python3 python3-pip && pip install pipenv
```
Setting up the server
---------------------
2. Install required Python packages:
1. Install Python 3 and pip:
```sh
pipenv sync
```
# pacman -S python python-pip
# # OR:
# apt-get install python3 python3-pip
3. Set up Celery message broker. Pick a Celery backend (Redis or RabbitMQ are recommended), and install and enable the required packages. For example:
2. Install required Python packages:
```sh
# Redis support requires an additional package
pipenv run pip install redis
sudo pacman -S redis # or sudo apt-get install redis
sudo systemctl enable --now redis
```
# # You might have to use pip3 if your system defaults to Python 2
# pip install -r requirements.txt
4. Configure the server by adding options to `config.py`.
See `server_list/config.py` for defaults.
3. If using in production, install uwsgi and it's python plugin:
5. Start the server for development:
# pacman -S uwsgi uwsgi-plugin-python
# # OR:
# apt-get install uwsgi uwsgi-plugin-python
# # OR:
# pip install uwsgi
```sh
pipenv run flask run
```
4. Install, start, and enable MongoDB on boot:
6. Start the celery background worker:
# pacman -S mongodb && systemctl enable mongodb --now
```sh
pipenv run celery --app server_list:celery worker --beat
```
5. Configure the server by adding options to `config.py`.
See `config-example.py` for defaults.
Running in Production
---
6. Start the server:
When running in production you should set up a proxy server that calls the server list through WSGI.
$ ./server.py
$ # Or for production:
$ uwsgi -s /tmp/minetest-master.sock --plugin python -w server:app --enable-threads
$ # Then configure according to http://flask.pocoo.org/docs/deploying/uwsgi/
These examples assume that the server list is installed to `/srv/http/serverlist`.
7. (optional) Configure the proxy server, if any. You should make the server
load static files directly from the static directory. Also, `/list`
should be served from `list.json`. Example for nginx:
### Nginx
First [set up uWSGI](#uwsgi), then update the Nginx configuration to proxy to uWSGI. You should make the server load static files directly from the static directory. Also, `/list` should be aliased to `list.json`.
Here's an example configuration:
```nginx
root /srv/http/serverlist/server_list/static;
rewrite ^/list$ /list.json;
try_files $uri @uwsgi;
location @uwsgi {
uwsgi_pass unix:/run/uwsgi/server_list.sock;
}
```
Also see [the Flask uwsgi documentation](https://flask.palletsprojects.com/en/2.0.x/deploying/uwsgi/).
### Apache
There are two options for Apache, you can use either `mod_wsgi` or `mod_proxy_uwsgi`.
Note: both of these example configurations serve static through WSGI, instead of bypassing WSGI for performance.
#### mod_wsgi
First install/enable `mod_wsgi`.
Then create `wsgi.py` in the directory containing `server_list` with the following contents:
```py
import os, sys
sys.path.append(os.path.dirname(__file__))
from server_list import app
```
Then configure the Apache VirtualHost like the following:
```apache
WSGIDaemonProcess server_list python-home=<output of pipenv --venv>
WSGIProcessGroup server_list
WSGIApplicationGroup %{GLOBAL}
WSGIScriptAlias / /srv/http/serverlist/wsgi.py
WSGICallableObject app
<Directory /srv/http/serverlist>
<Files wsgi.py>
Require all granted
</Files>
</Directory>
```
#### mod_proxy_uwsgi
First [set up uWSGI](#uwsgi), then install/enable `mod_proxy` and `mod_proxy_uwsgi` and add the following to your VirtualHost:
```apache
ProxyPass / unix:/run/uwsgi/server_list.sock|uwsgi://localhost/
```
Note: this requires at least Apache 2.4.7 for the unix socket syntax. If you have an older version of Apache you'll have to use IP sockets.
### uWSGI
First, install uWSGI and its python plugin.
```sh
pacman -S uwsgi uwsgi-plugin-python
# OR:
apt-get install uwsgi uwsgi-plugin-python
# OR:
pip install uwsgi
```
Then create a uWSGI config file. For example:
```ini
[uwsgi]
socket = /run/uwsgi/server_list.sock
plugin = python
virtualenv = <output of pipenv --venv>
python-path = /srv/http/serverlist
module = server_list
callable = app
```
You can put the config file in `/etc/uwsgi/server_list.ini`. Make sure that uWSGI is configured to start as the appropriate user and group for your distro (e.g. http:http) and then start and enable uWSGI.
```sh
systemctl enable --now uwsgi@server_list.service
```
root /path/to/server/static;
rewrite ^/list$ /list.json;
try_files $uri @uwsgi;
location @uwsgi {
uwsgi_pass ...;
}
License
---
-------
The Minetest server list code is licensed under the GNU Lesser General Public
The Minetest master server is licensed under the GNU Lesser General Public
License version 2.1 or later (LGPLv2.1+). A LICENSE.txt file should have been
supplied with your copy of this software containing a copy of the license.

31
config-example.py Normal file
View File

@@ -0,0 +1,31 @@
from datetime import timedelta
# Enables detailed tracebacks and an interactive Python console on errors.
# Never use in production!
DEBUG = False
# Address for development server to listen on
HOST = "127.0.0.1"
# Port for development server to listen on
PORT = 5000
# Amount of time after which servers are removed from the list if they haven't
# updated their listings. Note: By default Minetest servers only announce
# once every 5 minutes, so this should be more than that.
UPDATE_TIME = timedelta(minutes=6)
# Amount of time after which servers are removed from the database if they
# haven't updated their listings.
PURGE_TIME = timedelta(days=30)
# Creates server entries if a server sends an 'update' and there is no entry yet.
# This should only be used to populate the server list after list.json was deleted.
# This WILL cause problems such as mapgen, mods and privilege information missing from the list
ALLOW_UPDATE_WITHOUT_OLD = False
# Number of days' data to factor into popularity calculation
POP_DAYS = 3
# Address of the MongoDB server. You can use domain sockets on unix.
MONGO_URI = "mongodb://localhost/minetest-master"

40
migrations/alembic.ini
View File

@@ -1,40 +0,0 @@
[alembic]
[loggers]
keys = root,sqlalchemy,alembic,flask_migrate
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[logger_flask_migrate]
level = INFO
handlers =
qualname = flask_migrate
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

76
migrations/env.py
View File

@@ -1,76 +0,0 @@
from __future__ import with_statement
import logging
from logging.config import fileConfig
from flask import current_app
from alembic import context
config = context.config
fileConfig(config.config_file_name)
logger = logging.getLogger('alembic.env')
config.set_main_option(
'sqlalchemy.url',
str(current_app.extensions['migrate'].db.get_engine().url).replace('%', '%%'))
target_metadata = current_app.extensions['migrate'].db.metadata
def run_migrations_offline():
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.get_main_option("sqlalchemy.url")
context.configure(
url=url, target_metadata=target_metadata, literal_binds=True
)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online():
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
# this callback is used to prevent an auto-migration from being generated
# when there are no changes to the schema
# reference: http://alembic.zzzcomputing.com/en/latest/cookbook.html
def process_revision_directives(context, revision, directives):
if getattr(config.cmd_opts, 'autogenerate', False):
script = directives[0]
if script.upgrade_ops.is_empty():
directives[:] = []
logger.info('No changes in schema detected.')
connectable = current_app.extensions['migrate'].db.get_engine()
with connectable.connect() as connection:
context.configure(
connection=connection,
target_metadata=target_metadata,
process_revision_directives=process_revision_directives,
**current_app.extensions['migrate'].configure_args
)
with context.begin_transaction():
context.run_migrations()
if context.is_offline_mode():
run_migrations_offline()
else:
run_migrations_online()

22
migrations/script.py.mako
View File

@@ -1,22 +0,0 @@
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision | comma,n}
Create Date: ${create_date.date()}
"""
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade():
${upgrades if upgrades else "pass"}
def downgrade():
${downgrades if downgrades else "pass"}

73
migrations/versions/00ac5d537063_initial_migration.py
View File

@@ -1,73 +0,0 @@
"""Initial migration
Revision ID: 00ac5d537063
Create Date: 2021-06-12
"""
from alembic import op
import sqlalchemy as sa
revision = '00ac5d537063'
down_revision = None
branch_labels = None
depends_on = None
def upgrade():
op.create_table('server',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('world_uuid', sa.String(length=36), nullable=True),
sa.Column('online', sa.Boolean(), nullable=False),
sa.Column('address', sa.String(), nullable=False),
sa.Column('port', sa.Integer(), nullable=False),
sa.Column('announce_ip', sa.String(), nullable=False),
sa.Column('server_id', sa.String(), nullable=True),
sa.Column('clients', sa.String(), nullable=True),
sa.Column('clients_top', sa.Integer(), nullable=False),
sa.Column('clients_max', sa.Integer(), nullable=False),
sa.Column('first_seen', sa.DateTime(), nullable=False),
sa.Column('start_time', sa.DateTime(), nullable=False),
sa.Column('last_update', sa.DateTime(), nullable=False),
sa.Column('total_uptime', sa.Float(), nullable=False),
sa.Column('down_time', sa.DateTime(), nullable=True),
sa.Column('game_time', sa.Integer(), nullable=False),
sa.Column('lag', sa.Float(), nullable=True),
sa.Column('ping', sa.Float(), nullable=False),
sa.Column('mods', sa.String(), nullable=True),
sa.Column('version', sa.String(), nullable=False),
sa.Column('proto_min', sa.Integer(), nullable=False),
sa.Column('proto_max', sa.Integer(), nullable=False),
sa.Column('game_id', sa.String(), nullable=False),
sa.Column('mapgen', sa.String(), nullable=True),
sa.Column('url', sa.String(), nullable=True),
sa.Column('default_privs', sa.String(), nullable=True),
sa.Column('name', sa.String(), nullable=False),
sa.Column('description', sa.String(), nullable=False),
sa.Column('popularity', sa.Float(), nullable=False),
sa.Column('geo_continent', sa.String(length=2), nullable=True),
sa.Column('creative', sa.Boolean(), nullable=False),
sa.Column('is_dedicated', sa.Boolean(), nullable=False),
sa.Column('damage_enabled', sa.Boolean(), nullable=False),
sa.Column('pvp_enabled', sa.Boolean(), nullable=False),
sa.Column('password_required', sa.Boolean(), nullable=False),
sa.Column('rollback_enabled', sa.Boolean(), nullable=False),
sa.Column('can_see_far_names', sa.Boolean(), nullable=False),
sa.PrimaryKeyConstraint('id')
)
op.create_index('ix_server_address_port', 'server', ['address', 'port'], unique=True)
op.create_index(op.f('ix_server_online'), 'server', ['online'], unique=False)
op.create_index(op.f('ix_server_world_uuid'), 'server', ['world_uuid'], unique=True)
op.create_table('stats',
sa.Column('id', sa.Integer(), nullable=False),
sa.Column('max_servers', sa.Integer(), nullable=False),
sa.Column('max_clients', sa.Integer(), nullable=False),
sa.PrimaryKeyConstraint('id')
)
def downgrade():
op.drop_table('stats')
op.drop_index(op.f('ix_server_world_uuid'), table_name='server')
op.drop_index(op.f('ix_server_online'), table_name='server')
op.drop_index('ix_server_address_port', table_name='server')
op.drop_table('server')

23
migrations/versions/d6af394ec1ab_add_address_verification_required_field.py
View File

@@ -1,23 +0,0 @@
"""Add address verification required field
Revision ID: d6af394ec1ab
Revises: 00ac5d537063
Create Date: 2021-07-10
"""
from alembic import op
import sqlalchemy as sa
from sqlalchemy.sql.expression import text
revision = 'd6af394ec1ab'
down_revision = '00ac5d537063'
branch_labels = None
depends_on = None
def upgrade():
op.add_column('server', sa.Column('address_verification_required', sa.Boolean(), nullable=False, server_default=text('0')))
def downgrade():
op.drop_column('server', 'address_verification_required')

6
package.json
View File

@@ -1,6 +0,0 @@
{
"dependencies": {
"commander": "^7.2.0",
"dot": "^1.1.3"
}
}

9
requirements.txt
View File

@@ -1,5 +1,4 @@
maxminddb-geolite2>=2018.703
Flask~=2.0
flask-sqlalchemy~=2.0
flask-migrate~=3.0
celery~=5.0
APScheduler>=3
Flask>=0.10
Flask-PyMongo>=0.3

428
server.py Executable file
View File

@@ -0,0 +1,428 @@
#!/usr/bin/env python3
import os, sys, json, time, socket
from datetime import date, datetime, timedelta
from threading import Thread
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask, request, send_from_directory
from flask.ext.pymongo import PyMongo, ASCENDING, DESCENDING
# Set up scheduler
sched = BackgroundScheduler(timezone="UTC")
sched.start()
app = Flask(__name__, static_url_path = "")
# Load configuration
app.config.from_pyfile("config-example.py") # Use example for defaults
if os.path.isfile(os.path.join(app.root_path, "config.py")):
app.config.from_pyfile("config.py")
# Set up database
mongo = PyMongo(app)
# Views
@app.route("/")
def index():
return app.send_static_file("index.html")
@app.route("/list")
def list():
# We have to make sure that the list isn't cached,
# since the list isn't really static.
return send_from_directory(app.static_folder, "list.json",
cache_timeout=0)
@app.route("/announce", methods=["GET", "POST"])
def announce():
ip = request.remote_addr
if ip.startswith("::ffff:"):
ip = ip[7:]
data = request.values["json"]
if len(data) > 5000:
return "JSON data is too big.", 413
try:
server = json.loads(data)
except:
return "Unable to process JSON data.", 400
if type(server) != dict:
return "JSON data is not an object.", 400
if not "action" in server:
return "Missing action field.", 400
action = server["action"]
if action not in ("start", "update", "delete"):
return "Invalid action field.", 400
server["ip"] = ip
if not "port" in server:
server["port"] = 30000
#### Compatability code ####
# port was sent as a string instead of an integer
elif type(server["port"]) == str:
server["port"] = int(server["port"])
#### End compatability code ####
old = serverList.get(ip, server["port"])
if action == "delete":
if not old:
return "Server not found.", 500
serverList.remove(old)
serverList.save()
return "Removed from server list."
elif not checkRequest(server):
return "Invalid JSON data.", 400
if action == "update" and not old:
if app.config["ALLOW_UPDATE_WITHOUT_OLD"]:
old = server
old["start"] = time.time()
old["clients_top"] = 0
old["updates"] = 0
old["total_clients"] = 0
else:
return "Server to update not found.", 500
server["last_update"] = datetime.utcnow()
del server["uptime"]
server["start"] = time.time() if action == "start" else old["start"]
if "clients_list" in server:
server["clients"] = len(server["clients_list"])
server["clients_top"] = max(server["clients"], old["clients_top"]) if old else server["clients"]
# Make sure that startup options are saved
if action == "update":
for field in ("dedicated", "rollback", "mapgen", "privs",
"can_see_far_names", "mods"):
if field in old:
server[field] = old[field]
## Popularity
# This should only include stats from the past few days, so storing
# the necessary data is fairly complicated. Three pieces of
# information are required: the time the server's been connected
# (number of updates), the number of clients reported, and a
# timestamp so that this information can expire. This is stored in
# the format {date: (updates, clients)}.
total_updates = 1
# This is the total of all the client numbers we've received,
# it includes clients that were on in previous updates.
total_clients = server["clients"]
today = date.today()
# Key must be a string
today_key = str(today.toordinal())
if old and type(old["updates"]) == dict:
server["updates"] = {}
updates = server["updates"]
# Copy over only recent updates, and set the total counter
pop_days = app.config["POP_DAYS"]
for d, data in old["updates"].items():
if date.fromordinal(int(d)) >= \
today - timedelta(days=pop_days):
updates[d] = data
total_updates += data[0]
total_clients += data[1]
if today_key in updates:
updates[today_key][0] += 1
updates[today_key][1] += server["clients"]
else:
updates[today_key] = (1, server["clients"])
else:
server["updates"] = {today_key: (1, server["clients"])}
server["pop_v"] = total_clients / total_updates
# Keep server record ID
if old:
server["_id"] = old["_id"]
finishRequestAsync(server)
return "Thanks, your request has been filed.", 202
# Utilities
# Returns ping time in seconds (up), False (down), or None (error).
def serverUp(info):
try:
sock = socket.socket(info[0], info[1], info[2])
sock.settimeout(3)
sock.connect(info[4])
buf = b"\x4f\x45\x74\x03\x00\x00\x00\x01"
sock.send(buf)
start = time.time()
data = sock.recv(1024)
end = time.time()
if not data:
return False
peer_id = data[12:14]
buf = b"\x4f\x45\x74\x03" + peer_id + b"\x00\x00\x03"
sock.send(buf)
sock.close()
return end - start
except socket.timeout:
return False
except:
return None
# fieldName: (Required, Type, SubType)
fields = {
"action": (True, "str"),
"address": (False, "str"),
"port": (False, "int"),
"clients": (True, "int"),
"clients_max": (True, "int"),
"game_time": (True, "int"),
"lag": (False, "float"),
"clients_list": (False, "list", "str"),
"mods": (False, "list", "str"),
"version": (True, "str"),
"proto_min": (False, "int"),
"proto_max": (False, "int"),
"gameid": (True, "str"),
"mapgen": (False, "str"),
"url": (False, "str"),
"privs": (False, "str"),
"name": (True, "str"),
"description": (True, "str"),
# Flags
"creative": (False, "bool"),
"dedicated": (False, "bool"),
"damage": (False, "bool"),
"liquid_finite": (False, "bool"),
"pvp": (False, "bool"),
"password": (False, "bool"),
"rollback": (False, "bool"),
"can_see_far_names": (False, "bool"),
}
def checkRequest(server):
for name, data in fields.items():
if not name in server:
if data[0]: return False
else: continue
#### Compatibility code ####
# Accept strings in boolean fields but convert it to a
# boolean, because old servers sent some booleans as strings.
if data[1] == "bool" and type(server[name]).__name__ == "str":
server[name] = True if server[name].lower() in ("true", "1") else False
continue
# clients_max was sent as a string instead of an integer
if name == "clients_max" and type(server[name]).__name__ == "str":
server[name] = int(server[name])
continue
#### End compatibility code ####
if type(server[name]).__name__ != data[1]:
return False
if len(data) >= 3:
for item in server[name]:
if type(item).__name__ != data[2]:
return False
if "_id" in server:
return False
return True
def finishRequestAsync(server):
th = Thread(name = "ServerListThread",
target = asyncFinishThread,
args = (server,))
th.start()
def asyncFinishThread(server):
checkAddress = False
if not "address" in server or not server["address"]:
server["address"] = server["ip"]
else:
checkAddress = True
try:
info = socket.getaddrinfo(server["address"],
server["port"],
type=socket.SOCK_DGRAM,
proto=socket.SOL_UDP)
except socket.gaierror:
app.logger.warning("Unable to get address info for %s." % (server["address"],))
return
if checkAddress:
addresses = set(data[4][0] for data in info)
if not server["ip"] in addresses:
app.logger.warning("Invalid IP %s for address %s (address valid for %s)."
% (server["ip"], server["address"], addresses))
return
server["ping"] = serverUp(info[0])
if not server["ping"]:
app.logger.warning("Server %s:%d has no ping."
% (server["address"], server["port"]))
return
del server["action"]
with app.app_context():
serverList.updateServer(server)
class ServerList:
def __init__(self):
self.last_cache_update = 0
with app.app_context():
mongo.db.meta.create_index("key", unique=True, name="key")
self.info = mongo.db.meta.find_one({"key": "list_info"}) or {}
mongo.db.servers.create_index([("ip", ASCENDING), ("port", ASCENDING)],
unique=True,
name="address")
mongo.db.servers.create_index("points", name="points")
mongo.db.servers.create_index("last_update",
expireAfterSeconds=app.config["PURGE_TIME"].total_seconds(),
name="expiration")
if "max_servers" not in self.info:
self.info["max_servers"] = 0
if "max_clients" not in self.info:
self.info["max_clients"] = 0
def __del__(self):
with app.app_context():
self.updateMeta()
def getPoints(self, server):
points = 0
# 1 per client without a guest or all-numeric name.
if "clients_list" in server:
for name in server["clients_list"]:
if not name.startswith("Guest") and \
not name.isdigit():
points += 1
else:
# Old server
points = server["clients"] / 4
# Penalize highly loaded servers to improve player distribution.
# Note: This doesn't just make more than 16 players stop
# increasing your points, it can actually reduce your points
# if you have guests/all-numerics.
if server["clients"] > 16:
points -= server["clients"] - 16
# 1 per month of age, limited to 8
points += min(8, server["game_time"] / (60*60*24*30))
# 1/2 per average client, limited to 4
points += min(4, server["pop_v"] / 2)
# -8 for unrealistic max_clients
if server["clients_max"] >= 128:
points -= 8
# -8 per second of ping over 0.4s
if server["ping"] > 0.4:
points -= (server["ping"] - 0.4) * 8
# Up to -8 for less than an hour of uptime (penalty linearly decreasing)
HOUR_SECS = 60 * 60
uptime = server["uptime"]
if uptime < HOUR_SECS:
points -= ((HOUR_SECS - uptime) / HOUR_SECS) * 8
return points
def updateCache(self):
servers = mongo.db.servers.find({"last_update":
{"$gt": datetime.utcnow() - app.config["UPDATE_TIME"]}
}).sort("points", DESCENDING)
server_list = []
num_clients = 0
for server in servers:
del server["_id"]
del server["last_update"]
del server["updates"]
server["uptime"] = time.time() - server["start"]
server_list.append(server)
num_clients += server["clients"]
info = self.info
info["max_servers"] = max(len(server_list), info["max_servers"])
info["max_clients"] = max(num_clients, info["max_clients"])
with open(os.path.join("static", "list.json"), "w") as fd:
json.dump({
"total": {"servers": len(server_list), "clients": num_clients},
"total_max": {"servers": info["max_servers"], "clients": info["max_clients"]},
"list": server_list
},
fd,
indent = "\t" if app.config["DEBUG"] else None
)
self.updateMeta()
self.last_cache_update = time.time()
# Update if servers are likely to have expired
def updateCacheIfNeeded(self):
if time.time() - self.last_cache_update < \
app.config["UPDATE_TIME"].total_seconds() / 2:
return
with app.app_context():
self.updateCache()
def updateMeta(self):
if not "key" in self.info:
self.info["key"] = "list_info"
mongo.db.meta.replace_one({"key": "list_info"},
self.info, upsert=True)
def get(self, ip, port):
return mongo.db.servers.find_one({"ip": ip, "port": port})
def remove(self, server):
mongo.db.servers.delete_one({"_id": server["_id"]})
self.updateCache()
def updateServer(self, server):
server["points"] = self.getPoints(server)
if "_id" in server:
mongo.db.servers.replace_one({"_id": server["_id"]},
server)
else:
mongo.db.servers.insert_one(server)
self.updateCache()
serverList = ServerList()
sched.add_job(serverList.updateCacheIfNeeded, "interval",
seconds=app.config["UPDATE_TIME"].total_seconds() / 2, coalesce=True,
max_instances=1)
if __name__ == "__main__":
serverList.updateCacheIfNeeded()
app.run(host = app.config["HOST"], port = app.config["PORT"])

2
server_list/__init__.py
View File

@@ -1,2 +0,0 @@
from .app import app, celery
from . import commands, tasks, views

35
server_list/app.py
View File

@@ -1,35 +0,0 @@
import os
from celery import Celery
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_migrate import Migrate
app = Flask(__name__, static_url_path="")
# Load defaults
app.config.from_pyfile("config.py")
# Load configuration
if os.path.isfile(os.path.join(app.root_path, "..", "config.py")):
app.config.from_pyfile("../config.py")
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
migrate = Migrate(app, db)
celery = Celery(
app.import_name,
broker=app.config['CELERY_BROKER_URL']
)
celery.conf.update(app.config)
class ContextTask(celery.Task):
def __call__(self, *args, **kwargs):
with app.app_context():
return self.run(*args, **kwargs)
celery.Task = ContextTask

35
server_list/commands.py
View File

@@ -1,35 +0,0 @@
import json
import click
from .app import app, db
from .models import Server, Stats
@app.cli.command("load-json")
@click.argument("filename")
@click.option("--update")
def load_json(filename, update):
"""Load the SQL database with servers from a JSON server list.
"""
with open(filename, "r") as fd:
data = json.load(fd)
assert data
for obj in data["list"]:
if update:
obj.setdefault("address", obj["ip"])
Server.create_or_update(obj)
else:
server = Server()
server.update(obj, True)
db.session.add(server)
stats = Stats.get()
stats.max_servers = data["total_max"]["servers"]
stats.max_clients = data["total_max"]["clients"]
db.session.add(stats)
db.session.commit()
click.echo(click.style(f'Loaded {len(data["list"])} servers', fg="green"))

45
server_list/config.py
View File

@@ -1,45 +0,0 @@
from datetime import timedelta
from glob import glob
# Enables detailed tracebacks and an interactive Python console on errors.
# Never use in production!
DEBUG = False
# Amount of time, in seconds, after which servers are removed from the list
# if they haven't updated their listings. Note: By default Minetest servers
# only announce once every 5 minutes, so this should be more than 300.
PURGE_TIME = timedelta(minutes=6)
# List of banned IP addresses for announce
# e.g. ['2620:101::44']
BANNED_IPS = []
# List of banned servers as host/port pairs
# e.g. ['1.2.3.4/30000', 'lowercase.hostname', 'lowercase.hostname/30001']
BANNED_SERVERS = []
# Creates server entries if a server sends an 'update' and there is no entry yet.
# This should only be used to populate the server list after list.json was deleted.
# This WILL cause problems such as mapgen, mods and privilege information missing from the list
ALLOW_UPDATE_WITHOUT_OLD = False
# Database to use to store persistent server information
SQLALCHEMY_DATABASE_URI = "sqlite:///server_list.sqlite"
# How strongly past player counts are weighted into the popularity
# over the current player count.
POPULARITY_FACTOR = 0.9
# Message broker to forward messages from web server to worker threads
# Redis and RabbitMQ are good options.
#CELERY_BROKER_URL = "redis://localhost/0"
# Maximum number of clients before a server will be considered heavily loaded
# and down-weighted to improve player distribution.
CLIENT_LIMIT = 32
# MaxMind GeoIP database.
# You can download a copy from https://db-ip.com/db/download/ip-to-country-lite
mmdbs = glob("dbip-country-lite-*.mmdb")
if mmdbs:
MAXMIND_DB = mmdbs[0]

270
server_list/models.py
View File

@@ -1,270 +0,0 @@
from datetime import datetime
from sqlalchemy.orm.exc import NoResultFound
from .app import app, db
class Server(db.Model):
__table_args__ = (db.Index("ix_server_address_port", "address", "port", unique=True),)
id = db.Column(db.Integer, primary_key=True)
# World-specific UUID used to identify the server.
# This is kept secret to prevent anyone from spoofing the server.
world_uuid = db.Column(db.String(36), nullable=True, index=True, unique=True)
# Whether the server is currently online
online = db.Column(db.Boolean, index=True, nullable=False, default=True)
# Server sent connection address
address = db.Column(db.String, nullable=False)
port = db.Column(db.Integer, nullable=False, default=30000)
# IP address announcement was received from
announce_ip = db.Column(db.String, nullable=False)
# Name of server software. E.g. "minetest"
server_id = db.Column(db.String, nullable=True)
# List of player names, one per line
clients = db.Column(db.String, nullable=True)
# Highest number of clients ever seen
clients_top = db.Column(db.Integer, nullable=False)
# Maximum number of allowed clients
clients_max = db.Column(db.Integer, nullable=False)
# First time that we received an announcement from this server
first_seen = db.Column(db.DateTime, default=datetime.utcnow, nullable=False)
# Time that server sent "start" announcement.
# This can be used to calculate the current uptime.
start_time = db.Column(db.DateTime, default=datetime.utcnow, nullable=False)
# Time of most recent update request
last_update = db.Column(db.DateTime, default=datetime.utcnow, nullable=False)
# Amount of time that we've seen the server up for, in seconds
total_uptime = db.Column(db.Float, nullable=False)
# Most recent time that the server went down
down_time = db.Column(db.DateTime, nullable=True)
# Server sent value for age of world.
# Should nearly match uptime on a server that always announces.
game_time = db.Column(db.Integer, nullable=False)
# Server sent value based on sever loop timing
lag = db.Column(db.Float, nullable=True)
# Ping time in seconds
ping = db.Column(db.Float, nullable=False)
# List of enabled mods, one per line
mods = db.Column(db.String, nullable=True)
# Server release version
version = db.Column(db.String, nullable=False)
# Supported protocol versions
proto_min = db.Column(db.Integer, nullable=False)
proto_max = db.Column(db.Integer, nullable=False)
game_id = db.Column(db.String, nullable=False)
# Mapgen name
mapgen = db.Column(db.String, nullable=True)
# Server landing page URL
url = db.Column(db.String, nullable=True)
# Privileges granted to new players by default
default_privs = db.Column(db.String, nullable=True)
name = db.Column(db.String, nullable=False)
description = db.Column(db.String, nullable=False)
# Roughly the average number of players on the server
popularity = db.Column(db.Float, nullable=False)
# Continent determined from IP
geo_continent = db.Column(db.String(2), nullable=True)
# Flags
creative = db.Column(db.Boolean, nullable=False)
is_dedicated = db.Column(db.Boolean, nullable=False)
damage_enabled = db.Column(db.Boolean, nullable=False)
pvp_enabled = db.Column(db.Boolean, nullable=False)
password_required = db.Column(db.Boolean, nullable=False)
rollback_enabled = db.Column(db.Boolean, nullable=False)
can_see_far_names = db.Column(db.Boolean, nullable=False)
address_verification_required = db.Column(db.Boolean, nullable=False, default=False)
@staticmethod
def find_from_json(obj):
try:
if "world_uuid" in obj:
return Server.query.filter_by(world_uuid=obj["world_uuid"]).one()
return Server.query.filter_by(address=obj["address"], port=obj["port"]).one()
except NoResultFound:
return None
@staticmethod
def create_or_update(obj):
server = Server.find_from_json(obj)
if server is not None:
server.update(obj)
else:
server = Server()
server.update(obj, True)
db.session.add(server)
return server
def update(self, obj, initial=False):
now = datetime.now()
action = obj.get("action", "start")
assert action != "delete"
if "clients_list" in obj:
num_clients = len(obj["clients_list"])
else:
num_clients = obj["clients"]
if initial:
# Values set only when the server is first created
assert action == "start"
self.world_uuid = obj.get("world_uuid")
self.clients_top = num_clients
self.total_uptime = 0
else:
self.clients_top = max(self.clients_top, num_clients)
if action == "start":
# Fields updated only on startup
self.start_time = now
self.mods = "\n".join(obj.get("mods", []))
self.mapgen = obj.get("mapgen")
self.default_privs = obj.get("privs")
self.is_dedicated = obj.get("dedicated", False)
self.rollback_enabled = obj.get("rollback", False)
self.can_see_far_names = obj.get("can_see_far_names", False)
self.online = True
self.address = obj["address"]
self.port = obj.get("port", 30000)
self.announce_ip = obj["ip"]
self.server_id = obj.get("server_id")
self.clients = "\n".join(obj["clients_list"])
self.clients_max = obj["clients_max"]
self.game_time = obj["game_time"]
self.lag = obj.get("lag")
self.ping = obj["ping"]
self.version = obj["version"]
self.proto_min = obj["proto_min"]
self.proto_max = obj["proto_max"]
self.game_id = obj["gameid"]
self.url = obj.get("url")
self.name = obj["name"]
self.description = obj["description"]
if initial:
self.popularity = num_clients
else:
pop_factor = app.config["POPULARITY_FACTOR"]
self.popularity = self.popularity * pop_factor + \
num_clients * (1 - pop_factor)
self.geo_continent = obj.get("geo_continent")
self.creative = obj.get("creative", False)
self.damage_enabled = obj.get("damage", False)
self.pvp_enabled = obj.get("pvp", False)
self.password_required = obj.get("password", False)
self.last_update = now
if obj["address_verified"]:
self.address_verification_required = True
def as_json(self):
obj = {
"address": self.address,
"can_see_far_names": self.can_see_far_names,
"clients_list": self.clients.split("\n") if self.clients else [],
"clients_max": self.clients_max,
"clients_top": self.clients_top,
"creative": self.creative,
"damage": self.damage_enabled,
"dedicated": self.is_dedicated,
"description": self.description,
"game_time": self.game_time,
"gameid": self.game_id,
"name": self.name,
"password": self.password_required,
"ping": self.ping,
"pop_v": self.popularity,
"port": self.port,
"proto_max": self.proto_max,
"proto_min": self.proto_min,
"pvp": self.pvp_enabled,
"rollback": self.rollback_enabled,
"uptime": int((datetime.utcnow() - self.start_time).total_seconds()),
"version": self.version,
}
# Optional fields
if self.geo_continent is not None:
obj["geo_continent"] = self.geo_continent
if self.lag is not None:
obj["lag"] = self.lag
if self.mapgen is not None:
obj["mapgen"] = self.mapgen
if self.mods is not None:
obj["mods"] = self.mods.split("\n") if self.mods else []
if self.default_privs is not None:
obj["privs"] = self.default_privs
if self.server_id is not None:
obj["server_id"] = self.server_id
if self.url is not None:
obj["url"] = self.url
return obj
def set_offline(self):
now = datetime.utcnow()
self.online = False
self.total_uptime += (now - self.start_time).total_seconds()
self.down_time = now
class Stats(db.Model):
"""
This table has only a single row storing all of the global statistics.
"""
id = db.Column(db.Integer, primary_key=True)
max_servers = db.Column(db.Integer, nullable=False, default=0)
max_clients = db.Column(db.Integer, nullable=False, default=0)
@staticmethod
def get():
try:
return Stats.query.filter_by(id=1).one()
except NoResultFound:
stats = Stats()
stats.id = 1
db.session.add(stats)
return stats

98
server_list/ping.py
View File

@@ -1,98 +0,0 @@
import time
import socket
from .app import app
from .util import get_addr_info
# Initial packet of type ORIGINAL, with no data.
# This should prompt the server to assign us a peer id.
# [0] u32 protocol_id (PROTOCOL_ID)
# [4] session_t sender_peer_id (PEER_ID_INEXISTENT)
# [6] u8 channel
# [7] u8 type (PACKET_TYPE_ORIGINAL)
PING_PACKET = b"\x4f\x45\x74\x03\x00\x00\x00\x01"
def get_ping_reply(data):
# [0] u32 protocol_id (PROTOCOL_ID)
# [4] session_t sender_peer_id
# [6] u8 channel
# [7] u8 type (PACKET_TYPE_RELIABLE)
# [8] u16 sequence number
# [10] u8 type (PACKET_TYPE_CONTROL)
# [11] u8 controltype (CONTROLTYPE_SET_PEER_ID)
# [12] session_t peer_id_new
peer_id = data[12:14]
# Send packet of type CONTROL, subtype DISCO,
# to cleanly close our server connection.
# [0] u32 protocol_id (PROTOCOL_ID)
# [4] session_t sender_peer_id
# [6] u8 channel
# [7] u8 type (PACKET_TYPE_CONTROL)
# [8] u8 controltype (CONTROLTYPE_DISCO)
return b"\x4f\x45\x74\x03" + peer_id + b"\x00\x00\x03"
def ping_server_addresses(address, port):
pings = []
addr_info = get_addr_info(address, port)
for record in addr_info:
ping = server_up(record)
if not ping:
app.logger.warning("Could not connect to %s:%d using resolved info %r.",
address, port, record)
return None
pings.append(ping)
return pings
def ping_server(sock):
sock.send(PING_PACKET)
# Receive reliable packet of type CONTROL, subtype SET_PEER_ID,
# with our assigned peer id as data.
start = time.time()
data = sock.recv(1024)
end = time.time()
if not data:
return None
sock.send(get_ping_reply(data))
return end - start
# Returns ping time in seconds (up) or None (down).
def server_up(info):
"""Pings a Minetest server to check if it is online.
"""
try:
sock = socket.socket(info[0], info[1], info[2])
sock.settimeout(2)
sock.connect(info[4])
except OSError:
return None
attempts = 0
pings = []
while len(pings) < 3 and attempts - len(pings) < 3:
attempts += 1
try:
ping = ping_server(sock)
if ping is not None:
pings.append(ping)
except socket.timeout:
pass
except ConnectionRefusedError:
return None
except OSError:
return None
sock.close()
if len(pings) != 0:
return min(pings)
return None

30
server_list/static/index.html
View File

@@ -1,30 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Minetest server list</title>
<style>
body {
font-family: Roboto, Ubuntu, "Segoe UI", sans-serif;
}
a {
color: #336B87;
}
a:visited {
color: #336BA1;
}
@media only screen and (max-width: 1024px) {
#server_list table .version, #server_list table .flags, #server_list table .uptime {
display: none;
}
}
</style>
<script>
var master = {show_proto_select: true};
</script>
</head>
<body>
<div id="server_list"></div>
<script src="list.js"></script>
</body>
</html>

83
server_list/static/servers.jst
View File

@@ -1,83 +0,0 @@
{{? !master.no_total}}
<div>
<span class="header_total">
Players: {{=it.total.clients}}/{{=it.total_max.clients}}&nbsp;
Servers: {{=it.total.servers}}/{{=it.total_max.servers}}
</span>
{{? master.show_proto_select}}
, Protocol: <select class="proto_select">
<option value="">All</option>
<option value="[11,32]" {{? master.proto_range=='[11,32]'}}selected{{?}}>11-32 (Minetest 0.4)</option>
<option value="[37,99]" {{? master.proto_range=='[37,99]'}}selected{{?}}>37+ (Minetest 5.x)</option>
</select>{{?}}
</div>
{{?}}
<table>
<thead><tr>
{{? !master.no_address}}<th>Address[:Port]</th>{{?}}
{{? !master.no_clients}}<th>Players / Max{{? !master.no_avgtop}}<br/>Average / Top{{?}}</th>{{?}}
{{? !master.no_version}}<th class="version">Version, Subgame[, Mapgen]</th>{{?}}
{{? !master.no_name}}<th>Name</th>{{?}}
{{? !master.no_description}}<th>Description</th>{{?}}
{{? !master.no_flags}}<th class="flags">Flags</th>{{?}}
{{? !master.no_uptime}}<th class="uptime">Uptime, Age</th>{{?}}
{{? !master.no_ping}}<th>Ping, Lag</th>{{?}}
</tr></thead>
<tbody>
{{~it.list :server:index}}
{{ if (master.limit && index + 1 > master.limit) break;}}
{{ if (master.min_clients && server.clients_list.length < master.min_clients) continue;}}
<tr>
{{? !master.no_address}}
<td class="address">
{{=addressString(server)}}
</td>{{?}}
{{? !master.no_clients}}
<td class="clients{{? server.clients_list.length > 0 }} mts_hover_list_text{{?}}">
{{=constantWidth(server.clients_list.length + '/' + server.clients_max, 3.4)}}
{{? !master.no_avgtop}} {{=constantWidth(Math.floor(server.pop_v) + '/' + server.clients_top, 3.4)}}{{?}}
{{=hoverList("Clients", server.clients_list)}}
</td>{{?}}
{{? !master.no_version}}
<td class="version{{? server.mods && server.mods.length > 0}} mts_hover_list_text{{?}}">
{{!server.version}}, {{!server.gameid}}
{{? server.mapgen}}, {{!server.mapgen}}{{?}}
{{=hoverList("Mods", server.mods)}}
</td>{{?}}
{{? !master.no_name}}
<td class="name">
{{? server.url}}
<a href="{{!server.url}}">{{=tooltipString(server.name)}}</a>
{{??}}
{{=tooltipString(server.name)}}
{{?}}
</td>{{?}}
{{? !master.no_description}}
<td class="description">
{{=tooltipString(server.description)}}
</td>{{?}}
{{? !master.no_flags}}
<td class="flags {{? server.privs}} mts_hover_list_text{{?}}">
{{=hoverString("Default privileges", server.privs)}}
{{=server.creative ? 'Cre ' : ''}}
{{=server.damage ? 'Dmg ' : ''}}
{{=server.pvp ? 'PvP ' : ''}}
{{=server.password ? 'Pwd ' : ''}}
{{=server.rollback ? 'Rol ' : ''}}
{{=server.can_see_far_names ? 'Far ' : ''}}
</td>{{?}}
{{? !master.no_uptime}}
<td class="uptime">
{{=constantWidth(humanTime(server.uptime), 3.2)}} / {{=constantWidth(humanTime(server.game_time), 3.2)}}
</td>{{?}}
{{? !master.no_ping}}
<td class="ping">
{{=constantWidth(Math.floor(server.ping * 1000), 1.8)}}{{? server.lag}} / {{=constantWidth(Math.floor(server.lag * 1000), 1.8)}}{{?}}
</td>{{?}}
</tr>
{{~}}
</tbody>
</table>
{{? master.min_clients || master.limit}}
<a href="javascript:delete master.min_clients; delete master.limit; get();">Show more...</a>
{{?}}

77
server_list/static/style.css
View File

@@ -1,77 +0,0 @@
#server_list .header_total {
font-weight: bold;
}
#server_list table {
max-width: 100%;
width: 100%;
background-color: transparent;
border-collapse: collapse;
border-spacing: 0;
font-size: small;
}
#server_list td, #server_list th {
border: 1px solid #2A3132;
padding: 5px;
color: inherit;
}
#server_list thead {
background-color: #2A3132;
border-bottom: 5px solid #336B87;
color: white;
font-size: 1.1em;
}
#server_list tbody tr:nth-child(even) {
background-color: #E0E0E0;
}
#server_list td.clients, #server_list td.uptime, #server_list td.ping {
text-align: center;
}
#server_list td.version, #server_list td.name, #server_list td.description {
text-overflow: ellipsis;
overflow: hidden;
white-space: nowrap;
}
/* Note: the column widths here are not exact as auto-layout is left enabled */
#server_list td.address {
max-width: 24ch;
}
#server_list td.version {
max-width: 16ch;
}
#server_list td.name {
max-width: 32ch;
}
#server_list td.description {
max-width: 64ch;
}
.mts_hover_list {
display: none;
border: 1px solid #336B87;
border-radius: 10px;
background-color: #FFF;
position: absolute;
z-index: 100;
padding: 0.5em;
box-shadow: 1px 1px 5px 3px rgba(0, 0, 0, 0.25);
}
.mts_hover_list b {
font-size: 1.1em;
}
td:hover .mts_hover_list {
display: block;
}
.mts_cwidth {
display: inline-block;
}

79
server_list/tasks.py
View File

@@ -1,79 +0,0 @@
import json
import os
from datetime import datetime
from .app import app, celery, db
from .models import Server, Stats
from .ping import ping_server_addresses
from .util import get_geo_continent, server_ranking
@celery.task
def update_server(obj):
geo_continent = get_geo_continent(obj["addr_info"][-1][4][0])
if geo_continent is not None:
obj["geo_continent"] = geo_continent
# Ensure that a Minetest server is actually reachable on all addresses
pings = ping_server_addresses(obj["address"], obj["port"])
if pings is None:
return
# Use average ping
obj["ping"] = sum(pings) / len(pings)
Server.create_or_update(obj)
db.session.commit()
def update_list_json():
online_servers = Server.query.filter_by(online=True).all()
online_servers.sort(key=server_ranking, reverse=True)
server_list = [s.as_json() for s in online_servers]
num_clients = 0
for server in server_list:
num_clients += len(server["clients_list"])
stats = Stats.get()
stats.max_servers = max(len(server_list), stats.max_servers)
stats.max_clients = max(num_clients, stats.max_clients)
list_path = os.path.join(app.static_folder, "list.json")
# Write to temporary file, then do an atomic replace so that clients don't
# see a truncated file if they load the list just as it's being updated.
with open(list_path + "~", "w") as fd:
debug = app.config["DEBUG"]
json.dump({
"total": {"servers": len(server_list), "clients": num_clients},
"total_max": {"servers": stats.max_servers, "clients": stats.max_clients},
"list": server_list,
},
fd,
indent="\t" if debug else None,
separators=(',', ': ') if debug else (',', ':')
)
os.replace(list_path + "~", list_path)
@celery.task
def update_list():
cutoff = datetime.utcnow() - app.config["PURGE_TIME"]
expired_servers = Server.query.filter(
Server.online == True,
Server.last_update < cutoff
)
for server in expired_servers:
server.set_offline()
update_list_json()
db.session.commit()
@celery.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
sender.add_periodic_task(60, update_list.s(), name='Update server list')

10
server_list/templates/address_verification_failed.txt
View File

@@ -1,10 +0,0 @@
Server address does not include a DNS record for the IP that the announcement was sent from.
Announce IP: {{ announce_ip }}
Address records: {{ valid_addresses | join ", " }}
Help: This is usually because your server is only listening on IPv4 but your announcement is being sent over IPv6.
If that is the case there are two ways to fix this:
1. (preferred) Set ipv6_server = true in your server config to listen on IPv6 and add your IPv6 address to DNS as an AAAA record.
On Linux this allows clients to connect using both IPv4 and IPv6 (unless you have enabled net.ipv6.bind6only).
On other operating systems this option may only work with IPv6 clients and you'll have to use the second option if you want to support IPv4.
2. Set bind_address = 0.0.0.0 in your server config to force IPv4 only, the announce will then be sent from the IPv4 address.

213
server_list/util.py
View File

@@ -1,213 +0,0 @@
import re
import socket
from datetime import datetime, timedelta
from .app import app
try:
import maxminddb
MAXMIND_DB = app.config.get("MAXMIND_DB", None)
if MAXMIND_DB is not None:
geoip_reader = maxminddb.open_database(MAXMIND_DB, maxminddb.MODE_AUTO)
else:
app.logger.warning(
"For working GeoIP download the database from "
"https://db-ip.com/db/download/ip-to-country-lite and point "
"the MAXMIND_DB setting to the .mmdb file."
)
geoip_reader = None
except ImportError:
app.logger.warning("maxminddb not available, GeoIP will not work.")
UUID_RE = re.compile('^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$')
def check_ban(announce_ip, address, port):
if "%s/%d" % (announce_ip, port) in app.config["BANNED_SERVERS"]:
return True
if address != announce_ip:
# Normalize address for ban checks
address = address.lower().rstrip(".")
if f"{address}/{port}" in app.config["BANNED_SERVERS"] or \
address in app.config["BANNED_SERVERS"]:
return True
return False
def get_addr_info(address, port):
try:
return socket.getaddrinfo(
address,
port,
type=socket.SOCK_DGRAM,
proto=socket.SOL_UDP)
except socket.gaierror:
app.logger.warning("Unable to get address info for [%s]:%d.",
address, port)
return None
def verify_announce(addr_info, address, announce_ip):
if address == announce_ip:
return True
addresses = set(data[4][0] for data in addr_info)
if not announce_ip in addresses:
app.logger.warning(
"Server address %r does not resolve to announce IP %r (address valid for %r).",
address, announce_ip, addresses)
return False
return True
def get_geo_continent(ip):
if ip.startswith("::ffff:"):
ip = ip[7:]
if reader is None:
return
try:
geo = geoip_reader.get(ip)
except ValueError:
return
if geo and "continent" in geo:
return geo["continent"]["code"]
else:
app.logger.warning("Unable to get GeoIP Continent data for %s.", ip)
return None
# fieldName: (Required, Type, SubType)
fields = {
"action": (True, "str"),
"world_uuid": (False, "str"),
"address": (False, "str"),
"port": (False, "int"),
"clients_max": (True, "int"),
"uptime": (True, "int"),
"game_time": (True, "int"),
"lag": (False, "float"),
"clients_list": (True, "list", "str"),
"mods": (False, "list", "str"),
"version": (True, "str"),
"proto_min": (True, "int"),
"proto_max": (True, "int"),
"gameid": (True, "str"),
"mapgen": (False, "str"),
"url": (False, "str"),
"privs": (False, "str"),
"name": (True, "str"),
"description": (True, "str"),
# Flags
"creative": (False, "bool"),
"dedicated": (False, "bool"),
"damage": (False, "bool"),
"pvp": (False, "bool"),
"password": (False, "bool"),
"rollback": (False, "bool"),
"can_see_far_names": (False, "bool"),
}
def check_request_json(obj):
"""Checks the types and values of fields in the request.
Returns error string or None.
"""
for name, data in fields.items():
# Delete optional string fields sent as empty strings
if not data[0] and data[1] == "str" and obj.get(name) == "":
del obj[name]
if not name in obj:
if data[0]:
return f"Required field '{name}' is missing."
continue
type_str = type(obj[name]).__name__
if type_str != data[1]:
return f"Field '{name}'' has incorrect type (expected {data[1]} found {type_str})."
if len(data) >= 3:
for item in obj[name]:
subtype_str = type(item).__name__
if subtype_str != data[2]:
return f"Entry in field '{name}' has incorrect type (expected {data[2]} found {subtype_str})."
if "url" in obj:
url = obj["url"]
if not any(url.startswith(p) for p in ["http://", "https://", "//"]):
return "Field 'url' does not match expected format."
if "world_uuid" in obj and not UUID_RE.match(obj["world_uuid"]):
return "Field 'world_uuid' does not match expected format."
return None
def server_ranking(server):
now = datetime.utcnow()
points = 0
clients = server.clients.split('\n')
# 1 per client, but only 1/8 per "guest" client
for name in server["clients_list"]:
if re.match(r"[A-Z][a-z]{3,}[1-9][0-9]{2,3}", name):
points += 1/8
else:
points += 1
# Penalize highly loaded servers to improve player distribution.
# Note: This doesn't just make more than 80% of max players stop
# increasing your points, it can actually reduce your points
# if you have guests.
cap = int(server.clients_max * 0.80)
if len(clients) > cap:
points -= len(clients) - cap
# 1 per month of age, limited to 8
points += min(8, (now - server.first_seen) / timedelta(months=1))
# 1/2 per average client, limited to 4
points += min(4, server.popularity / 2)
# -8 for unrealistic max_clients
if server.clients_max > 200:
points -= 8
# -8 per second of ping over 0.4s
if server.ping > 0.4:
points -= (server.ping - 0.4) * 8
# Up to -8 for less than an hour of uptime (penalty linearly decreasing)
ONE_HOUR = timedelta(hours=1)
uptime = now - server.start_time
if uptime < ONE_HOUR:
# Only apply penalty if the server was down for more than an hour
down_too_long = True
if server.down_time is not None:
down_too_long = (server.start_time - server.down_time) > ONE_HOUR
if down_too_long:
points -= ((ONE_HOUR - uptime) / ONE_HOUR) * 8
# Reduction to 40% for servers that support both legacy (v4) and v5 clients
if server.proto_min <= 32 and server.proto_max > 36:
points *= 0.4
return points

109
server_list/views.py
View File

@@ -1,109 +0,0 @@
import json
from flask import render_template, request, send_from_directory, make_response
from .app import app, db
from .models import Server
from .tasks import update_server
from .util import check_ban, check_request_json, get_addr_info, get_geo_continent, verify_announce
@app.route("/")
def index():
return app.send_static_file("index.html")
@app.route("/list")
def server_list():
# We have to make sure that the list isn't cached,
# since the list isn't really static.
return send_from_directory(app.static_folder, "list.json", max_age=0)
@app.route("/geoip")
def geoip():
continent = get_geo_continent(request.remote_addr)
resp = make_response({
"continent": continent, # null on error
})
resp.cache_control.max_age = 7 * 86400
resp.cache_control.private = True
return resp
@app.route("/announce", methods=["GET", "POST"])
def announce():
announce_ip = request.remote_addr
if announce_ip.startswith("::ffff:"):
announce_ip = announce_ip[7:]
if announce_ip in app.config["BANNED_IPS"]:
return "Banned.", 403
data = request.values["json"]
if len(data) > 8192:
return "JSON data is too big.", 413
try:
obj = json.loads(data)
except json.JSONDecodeError as e:
return "Failed to decode JSON: " + e.msg, 400
if not isinstance(obj, dict):
return "JSON data is not an object.", 400
action = obj.get("action")
if action not in ("start", "update", "delete"):
return "Action field is invalid or missing.", 400
obj["ip"] = announce_ip
if not obj.get("address"):
obj["address"] = announce_ip
obj.setdefault("port", 30000)
if check_ban(announce_ip, obj["address"], obj["port"]):
return "Banned", 403
server = Server.find_from_json(obj)
if action == "delete":
if not server:
return "Server not found."
server.set_offline()
db.session.commit()
return "Removed from server list."
# Delete message does not require most fields
error_str = check_request_json(obj)
if error_str is not None:
return "Invalid JSON data: " + error_str, 400
if action == "update" and not server:
if app.config["ALLOW_UPDATE_WITHOUT_OLD"]:
action = "start"
else:
return "Server to update not found.", 404
addr_info = get_addr_info(obj["address"], obj["port"])
if addr_info is None:
return f"Failed to resolve server address {obj['address']!r}.", 400
valid = False
if "world_uuid" not in obj:
valid = verify_announce(addr_info, obj["address"], obj["ip"])
if not valid and server and server.address_verification_required:
return render_template("address_verification_failed.txt",
announce_ip=announce_ip,
valid_addresses=[data[4][0] for data in addr_info]), 400
obj["address_verified"] = valid
obj["addr_info"] = addr_info
update_server.delay(obj)
return "Done.", 202

11
static/index.html Normal file
View File

@@ -0,0 +1,11 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>Minetest server list</title>
</head>
<body>
<div id="server_list"></div>
</body>
</html>
<script src="list.js"></script>

61
server_list/static/list.js → static/list.js
View File

@@ -5,9 +5,6 @@ if (!master.output) master.output = '#server_list';
if (!master.list) master.list = "list";
if (!master.list_root) master.list_root = master.root;
if (!master.list_url) master.list_url = master.list_root + master.list;
master.cached_json = null;
// Utility functions used by the templating code
function humanTime(seconds) {
if (typeof(seconds) != "number") return '?';
@@ -16,10 +13,10 @@ function humanTime(seconds) {
d: 86400,
h: 3600,
m: 60
};
}
for (var i in conv) {
if (seconds >= conv[i]) {
return (seconds / conv[i]).toFixed(i=='y'?1:0) + i;
return (seconds / conv[i]).toFixed(1) + i;
}
}
return seconds + 's';
@@ -40,22 +37,28 @@ function addressString(server) {
var str = '<span'
if (shortStr.length > 25) {
shortStr = shortStr.substr(0, 23) + "&hellip;";
str += ' title="' + addrStr + '"'
str += ' class="mts_tooltip" title="' + addrStr + '"'
}
if (server.port != 30000)
shortStr += ':' + server.port;
return str + '>' + shortStr + '</span>';
}
function tooltipString(str) {
function tooltipString(str, maxLen) {
str = escapeHTML(str);
return '<span title="' + str + '">' + str + '</div>';
var shortStr = str;
var ret = '<span';
if (shortStr.length > maxLen) {
shortStr = shortStr.substr(0, maxLen - 2) + "&hellip;";
ret += ' class="mts_tooltip" title="' + str + '"';
}
return ret + '>' + shortStr + '</span>';
}
function hoverList(name, list) {
if (!list || list.length == 0) return '';
var str = '<div class="mts_hover_list">'
str += '<b>' + name + '</b> (' + list.length + ')<br />';
str += name + ' (' + list.length + ')<br />';
for (var i in list) {
str += escapeHTML(list[i]) + '<br />';
}
@@ -64,50 +67,19 @@ function hoverList(name, list) {
function hoverString(name, string) {
if (!string) return '';
return '<div class="mts_hover_list">'
+ '<b>' + name + '</b>:<br />'
return '<div class="mts_hover_list">'
+ name + ':<br />'
+ escapeHTML(string) + '<br />'
+ '</div>';
}
function constantWidth(str, width) {
return '<span class="mts_cwidth" style="width:' + width + 'em;">' + str + '</span>';
}
// Code that fetches & displays the actual list
function draw(json) {
if (json == null)
return;
// pre-filter by chosen protocol range
var tmp = master.proto_range ? JSON.parse(master.proto_range) : null;
if (tmp) {
json = {
list: json.list.filter(function(server) {
return !(tmp[0] > server.proto_max || tmp[1] < server.proto_min);
}),
total: {clients: 0},
total_max: {clients: "?", servers: "?"}
};
json.list.forEach(function(server) { json.total.clients += server.clients; });
json.total.servers = json.list.length;
}
var html = window.render.servers(json);
jQuery(master.output).html(html);
jQuery('.proto_select', master.output).on('change', function(e) {
master.proto_range = e.target.value;
draw(master.cached_json); // re-render
});
}
function get() {
jQuery.getJSON(master.list_url, function(json) {
master.cached_json = json;
draw(json);
});
jQuery.getJSON(master.list_url, draw);
}
function loaded(){
@@ -117,7 +89,6 @@ function loaded(){
get();
}
// https://github.com/pyrsmk/toast
this.toast=function(){var e=document,t=e.getElementsByTagName("head")[0],n=this.setTimeout,r="createElement",i="appendChild",s="addEventListener",o="onreadystatechange",u="styleSheet",a=10,f=0,l=function(){--f},c,h=function(e,r,i,s){if(!t)n(function(){h(e)},a);else if(e.length){c=-1;while(i=e[++c]){if((s=typeof i)=="function"){r=function(){return i(),!0};break}if(s=="string")p(i);else if(i.pop){p(i[0]),r=i[1];break}}d(r,Array.prototype.slice.call(e,c+1))}},p=function(n,s){++f,/\.css$/.test(n)?(s=e[r]("link"),s.rel=u,s.href=n,t[i](s),v(s)):(s=e[r]("script"),s.src=n,t[i](s),s[o]===null?s[o]=m:s.onload=l)},d=function(e,t){if(!f)if(!e||e()){h(t);return}n(function(){d(e,t)},a)},v=function(e){if(e.sheet||e[u]){l();return}n(function(){v(e)},a)},m=function(){/ded|co/.test(this.readyState)&&l()};h(arguments)};
@@ -125,6 +96,6 @@ toast(master.root + 'style.css', master.root + 'servers.js', function() {
if (typeof(jQuery) != 'undefined')
return loaded();
else
toast('//ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js', loaded);
toast('//ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js', loaded);
});

76
static/servers.jst Normal file
View File

@@ -0,0 +1,76 @@
{{? !master.no_total}}
<div class="total">
Players: {{=it.total.clients}}/{{=it.total_max.clients}}&nbsp;
Servers: {{=it.total.servers}}/{{=it.total_max.servers}}
</div>
{{?}}
<table>
<thead><tr>
{{? !master.no_address}}<th>Address[:Port]</th>{{?}}
{{? !master.no_clients}}<th>Players / Max{{? !master.no_avgtop}}<br/>Average / Top{{?}}</th>{{?}}
{{? !master.no_version}}<th>Version, Subgame, Mapgenerator</th>{{?}}
{{? !master.no_name}}<th>Name</th>{{?}}
{{? !master.no_description}}<th>Description</th>{{?}}
{{? !master.no_flags}}<th>Flags</th>{{?}}
{{? !master.no_uptime}}<th>Uptime, Age</th>{{?}}
{{? !master.no_ping}}<th>Ping, Lag</th>{{?}}
</tr></thead>
<tbody>
{{~it.list :server:index}}
{{ if (master.limit && index + 1 > master.limit) break;}}
{{ if (master.min_clients && server.clients < master.min_clients) continue;}}
<tr>
{{? !master.no_address}}
<td class ="address">
{{=addressString(server)}}
</td>{{?}}
{{? !master.no_clients}}
<td class="clients{{? server.clients_list && server.clients_list.length > 0}} mts_hover_list_text{{?}}">
{{=server.clients}}/{{=server.clients_max}}{{? !master.no_avgtop}} &nbsp;&nbsp;{{=Math.floor(server.pop_v)}}/{{=server.clients_top}}{{?}}
{{=hoverList("Clients", server.clients_list)}}
</td>{{?}}
{{? !master.no_version}}
<td class="version{{? server.mods && server.mods.length > 0}} mts_hover_list_text{{?}}">
{{=escapeHTML(server.version)}}, {{=escapeHTML(server.gameid)}},&nbsp;
{{=escapeHTML(server.mapgen || '?')}}
{{=hoverList("Mods", server.mods)}}
</td>{{?}}
{{? !master.no_name}}
<td class="name">
{{? server.url}}
<a href="{{=escapeHTML(server.url)}}">{{=tooltipString(server.name, 25)}}</a>
{{??}}
{{=tooltipString(server.name, 25)}}
{{?}}
</td>{{?}}
{{? !master.no_description}}
<td class="description">
{{=tooltipString(server.description, 50)}}
</td>{{?}}
{{? !master.no_flags}}
<td class="flags {{? server.privs}} mts_hover_list_text{{?}}">
{{=hoverString("Privs", server.privs)}}
{{=server.creative ? 'Cre ' : ''}}
{{=server.dedicated ? 'Ded ' : ''}}
{{=server.damage ? 'Dmg ' : ''}}
{{=server.liquid_finite ? 'Liq ' : ''}}
{{=server.pvp ? 'PvP ' : ''}}
{{=server.password ? 'Pwd ' : ''}}
{{=server.rollback ? 'Rol ' : ''}}
{{=server.can_see_far_names ? 'Far ' : ''}}
</td>{{?}}
{{? !master.no_uptime}}
<td class="uptime">
{{=humanTime(server.uptime)}}, {{=humanTime(server.game_time)}}
</td>{{?}}
{{? !master.no_ping}}
<td class="ping">
{{=Math.floor(server.ping * 1000)}}{{? server.lag}}, {{= Math.floor(server.lag * 1000)}}{{?}}
</td>{{?}}
</tr>
{{~}}
</tbody>
</table>
{{? master.min_clients || master.limit}}
<a class="clickable" onclick="delete master.min_clients; delete master.limit; get();">More...</a>
{{?}}

53
static/style.css Normal file
View File

@@ -0,0 +1,53 @@
#server_list .total {
font-weight: bold;
}
#server_list table {
max-width: 100%;
width: 100%;
background-color: transparent;
border-collapse: collapse;
border-spacing: 0;
font-size: small;
}
#server_list td, #server_list th {
border: 1px solid gray;
}
#server_list thead {
background-color: #FFA;
}
#server_list tbody tr:nth-child(even) {
background-color: #EEE;
}
#server_list tbody tr:hover {
background-color: #CCC;
}
.mts_hover_list {
display: none;
border: 1px solid #88F;
border-radius: 4px;
background-color: white;
position: absolute;
z-index: 100;
padding: 0.5em;
}
td:hover .mts_hover_list {
display: block;
}
.mts_hover_list_text, .mts_tooltip {
text-decoration: underline;
text-decoration-style: dashed;
}
.clickable {
text-decoration: underline;
cursor: pointer;
}