Reduce chance of accidental release deletion

Fixes #176

.dockerignore

@@ -0,0 +1,5 @@
+.git
+data
+uploads
+*.pyc
+__pycache__

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,13 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: Unconfirmed Bug
+assignees: ''
+---
+
+## Summary
+Describe your problem here
+
+##### Steps to reproduce
+For bug reports or build issues, explain how the problem happened

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,25 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: Feature
+assignees: ''
+---
+
+## Problem
+
+A clear and concise description of what the problem is.
+ie: Why is this needed?
+Ex. I'm always frustrated when [...]
+
+## Solutions
+
+A clear and concise description of what you want to happen.
+
+## Alternatives
+
+A clear and concise description of any alternative solutions or features you've considered.
+
+## Additional context
+
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/policy.md

@@ -0,0 +1,7 @@
+---
+name: Policy suggestion
+about: Suggest a change to the guidelines
+title: ''
+labels: Policy
+assignees: ''
+---

.github/SECURITY.md

@@ -0,0 +1,19 @@
+# Security Policy
+
+## Supported Versions
+
+We only support the latest production version, deployed to <https://content.minetest.net>.
+See the [releases page](https://github.com/minetest/contentdb/releases).
+
+## Reporting a Vulnerability
+
+We ask that you report vulnerabilities privately, by contacting rubenwardy,
+to give us time to fix them. You can do that by using one of the methods outlined in the following link:
+
+* https://rubenwardy.com/contact/
+
+Depending on severity, we will either create a private issue for the vulnerability
+and release a security update, or give you permission to file the issue publicly.
+
+For more information on the justification of this policy, see
+[Responsible Disclosure](https://en.wikipedia.org/wiki/Responsible_disclosure).

.gitignore

@@ -1,11 +1,15 @@
-config.cfg
-config.prod.cfg
+/config.cfg
+/*.env
 *.sqlite
-main.css
+.vscode
+custom.css
 tmp
 log.txt
 *.rdb
-uploads
+app/public/uploads
+app/public/thumbnails
+celerybeat-schedule
+/data
 
 # Created by https://www.gitignore.io/api/linux,macos,python,windows
 

.gitlab-ci.yml

@@ -0,0 +1,22 @@
+image: docker/compose
+services:
+  - docker:dind
+cache:
+  key: "$CI_COMMIT_REF_SLUG"
+  paths:
+    - /var/lib/docker
+
+# build:
+#   stage: build
+#   script:
+#     - cp utils/gitlabci/* .
+#     - docker-compose build
+
+UI_Test:
+  stage: test
+  script:
+    - cp utils/gitlabci/* .
+    - docker-compose up -d
+    - ./utils/run_migrations.sh
+    - ./utils/tests_cov.sh
+    - docker-compose down

Dockerfile

@@ -0,0 +1,22 @@
+FROM python:3.6
+
+RUN groupadd -g 5123 cdb && \
+    useradd -r -u 5123 -g cdb cdb
+
+WORKDIR /home/cdb
+
+RUN mkdir /var/cdb
+RUN chown -R cdb:cdb /var/cdb
+
+COPY requirements.txt requirements.txt
+RUN pip install -r requirements.txt
+RUN pip install gunicorn
+
+COPY utils utils
+COPY config.cfg config.cfg
+COPY migrations migrations
+COPY app app
+
+RUN chown -R cdb:cdb /home/cdb
+
+USER cdb

README.md

@@ -1,58 +1,63 @@
 # Content Database
 
-## Setup
+Content database for Minetest mods, games, and more.
 
-First create a Python virtual env:
-
-	virtualenv env -ppython3
-	source env/bin/activate
-
-then use pip:
-
-	pip3 install -r requirements.txt
-
-### Development
-
-* Copy config.example.cfg to config.cfg
-* Fill SECRET_KEY and WTF_CSRF_SECRET_KEY in with a random string
-* Make a Github OAuth Client at <https://github.com/settings/developers>:
-	* Homepage URL - `http://localhost:5000/`
-	* Authorization callback URL - `http://localhost:5000/user/github/callback/`
-* Put client id and client secret in GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET
-* Setup the database: python3 setup.py
-
-
-## Running
-
-### Development
-
-You need to enter the virtual environment if you haven't yet in
-the current session:
-
-	source env/bin/activate
-
-If you need to, reset the db like so:
-
-	python3 setup.py -d
-
-Then run the server:
-
-	./rundebug.py
-
-Then view in your web browser: http://localhost:5000/
+Developed by rubenwardy, license GPLv3.0+.
 
 ## How-tos
 
-### Create migration
+Note: you should first read one of the guides on the [Github repo wiki](https://github.com/minetest/contentdb/wiki)
 
 ```sh
+# Run celery worker
+FLASK_CONFIG=../config.cfg celery -A app.tasks.celery worker
+
 # if sqlite
-python setup.py -t
+python utils/setup.py -t
 rm db.sqlite && python setup.py -t && FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db stamp head
 
 # Create migration
 FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db migrate
-
 # Run migration
-FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db migrate
+FLASK_CONFIG=../config.cfg FLASK_APP=app/__init__.py flask db upgrade
+
+# Enter docker
+docker exec -it contentdb_app_1 bash
+
+# Hot/live reload (only works with FLASK_DEBUG=1)
+./utils/reload.sh
+
+# Cold update a running version of CDB with minimal downtime
+./utils/update.sh
+```
+
+## Database
+
+
+```mermaid
+classDiagram
+
+User "1" --> "*" Package
+User --> UserEmailVerification
+User "1" --> "*" Notification
+Package "1" --> "*" Release
+Package "1" --> "*" Dependency
+Package "1" --> "*" Tag
+Package "1" --> "*" MetaPackage : provides
+Release --> MinetestVersion
+Package --> License
+Dependency --> Package
+Dependency --> MetaPackage
+MetaPackage "1" --> "*" Package
+Package "1" --> "*" Screenshot
+Package "1" --> "*" Thread
+Thread "1" --> "*" Reply
+Thread "1" --> "*" User : watchers
+User "1" --> "*" Thread
+User "1" --> "*" Reply
+User "1" --> "*" ForumTopic
+
+User --> "0..1" EmailPreferences
+User "1" --> "*" APIToken
+APIToken --> Package
 ```

app/__init__.py

@@ -17,25 +17,77 @@
 
 from flask import *
 from flask_user import *
+from flask_gravatar import Gravatar
 import flask_menu as menu
 from flask_mail import Mail
-from flask.ext import markdown
+from flaskext.markdown import Markdown
 from flask_github import GitHub
 from flask_wtf.csrf import CsrfProtect
 from flask_flatpages import FlatPages
-import os
+from flask_babel import Babel
+import os, redis
 
 app = Flask(__name__, static_folder="public/static")
 app.config["FLATPAGES_ROOT"] = "flatpages"
 app.config["FLATPAGES_EXTENSION"] = ".md"
 app.config.from_pyfile(os.environ["FLASK_CONFIG"])
 
+r = redis.Redis.from_url(app.config["REDIS_URL"])
+
 menu.Menu(app=app)
-markdown.Markdown(app, extensions=["fenced_code"], safe_mode=True, output_format="html5")
+markdown = Markdown(app, extensions=["fenced_code"], safe_mode=True, output_format="html5")
 github = GitHub(app)
 csrf = CsrfProtect(app)
 mail = Mail(app)
 pages = FlatPages(app)
+babel = Babel(app)
+gravatar = Gravatar(app,
+		size=58,
+		rating='g',
+		default='mp',
+		force_default=False,
+		force_lower=False,
+		use_ssl=True,
+		base_url=None)
 
-from . import models, tasks
-from .views import *
+from .sass import sass
+sass(app)
+
+
+if not app.debug and app.config["MAIL_UTILS_ERROR_SEND_TO"]:
+	from .maillogger import register_mail_error_handler
+	register_mail_error_handler(app, mail)
+
+
+@babel.localeselector
+def get_locale():
+	return request.accept_languages.best_match(app.config['LANGUAGES'].keys())
+
+from . import models, tasks, template_filters
+
+from .blueprints import create_blueprints
+create_blueprints(app)
+
+from flask_login import logout_user
+
+@app.route("/uploads/<path:path>")
+def send_upload(path):
+	return send_from_directory(app.config['UPLOAD_DIR'], path)
+
+@menu.register_menu(app, ".help", "Help", order=19, endpoint_arguments_constructor=lambda: { 'path': 'help' })
+@app.route('/<path:path>/')
+def flatpage(path):
+    page = pages.get_or_404(path)
+    template = page.meta.get('template', 'flatpage.html')
+    return render_template(template, page=page)
+
+@app.before_request
+def check_for_ban():
+	if current_user.is_authenticated:
+		if current_user.rank == models.UserRank.BANNED:
+			flash("You have been banned.", "error")
+			logout_user()
+			return redirect(url_for('user.login'))
+		elif current_user.rank == models.UserRank.NOT_JOINED:
+			current_user.rank = models.UserRank.MEMBER
+			models.db.session.commit()

app/blueprints/__init__.py

@@ -0,0 +1,10 @@
+import os, importlib
+
+def create_blueprints(app):
+	dir = os.path.dirname(os.path.realpath(__file__))
+	modules = next(os.walk(dir))[1]	
+	
+	for modname in modules:		
+		if all(c.islower() for c in modname):
+			module = importlib.import_module("." + modname, __name__)		
+			app.register_blueprint(module.bp)

app/blueprints/admin/__init__.py

@@ -15,12 +15,8 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
-from flask_user import current_user, login_required
-from app import app
-from app.models import *
+from flask import Blueprint
 
-@app.route("/notifications/")
-@login_required
-def notifications_page():
-    return render_template("notifications/list.html")
+bp = Blueprint("admin", __name__)
+
+from . import admin, licenseseditor, tagseditor, versioneditor

app/blueprints/admin/admin.py

@@ -0,0 +1,143 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+import flask_menu as menu
+from . import bp
+from app.models import *
+from celery import uuid, group
+from app.tasks.importtasks import importRepoScreenshot, importAllDependencies, makeVCSRelease, checkZipRelease
+from app.tasks.forumtasks  import importTopicList, checkAllForumAccounts
+from flask_wtf import FlaskForm
+from wtforms import *
+from app.utils import loginUser, rank_required, triggerNotif
+import datetime
+
+@bp.route("/admin/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def admin_page():
+	if request.method == "POST":
+		action = request.form["action"]
+		if action == "delstuckreleases":
+			PackageRelease.query.filter(PackageRelease.task_id != None).delete()
+			db.session.commit()
+			return redirect(url_for("admin.admin_page"))
+		elif action == "checkreleases":
+			releases = PackageRelease.query.filter(PackageRelease.url.like("/uploads/%")).all()
+
+			tasks = []
+			for release in releases:
+				zippath = release.url.replace("/uploads/", app.config["UPLOAD_DIR"])
+				tasks.append(checkZipRelease.s(release.id, zippath))
+
+			result = group(tasks).apply_async()
+
+			while not result.ready():
+				import time
+				time.sleep(0.1)
+
+			return redirect(url_for("todo.view"))
+		elif action == "importmodlist":
+			task = importTopicList.delay()
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("todo.topics")))
+		elif action == "checkusers":
+			task = checkAllForumAccounts.delay()
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
+		elif action == "importscreenshots":
+			packages = Package.query \
+				.filter_by(soft_deleted=False) \
+				.outerjoin(PackageScreenshot, Package.id==PackageScreenshot.package_id) \
+				.filter(PackageScreenshot.id==None) \
+				.all()
+			for package in packages:
+				importRepoScreenshot.delay(package.id)
+
+			return redirect(url_for("admin.admin_page"))
+		elif action == "restore":
+			package = Package.query.get(request.form["package"])
+			if package is None:
+				flash("Unknown package", "error")
+			else:
+				package.soft_deleted = False
+				db.session.commit()
+				return redirect(url_for("admin.admin_page"))
+		elif action == "importdepends":
+			task = importAllDependencies.delay()
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("admin.admin_page")))
+		elif action == "modprovides":
+			packages = Package.query.filter_by(type=PackageType.MOD).all()
+			mpackage_cache = {}
+			for p in packages:
+				if len(p.provides) == 0:
+					p.provides.append(MetaPackage.GetOrCreate(p.name, mpackage_cache))
+
+			db.session.commit()
+			return redirect(url_for("admin.admin_page"))
+		elif action == "recalcscores":
+			for p in Package.query.all():
+				p.setStartScore()
+
+			db.session.commit()
+			return redirect(url_for("admin.admin_page"))
+		elif action == "vcsrelease":
+			for package in Package.query.filter(Package.repo.isnot(None)).all():
+				if package.releases.count() != 0:
+					continue
+
+				rel = PackageRelease()
+				rel.package  = package
+				rel.title    = datetime.date.today().isoformat()
+				rel.url      = ""
+				rel.task_id  = uuid()
+				rel.approved = True
+				db.session.add(rel)
+				db.session.commit()
+
+				makeVCSRelease.apply_async((rel.id, "master"), task_id=rel.task_id)
+
+				msg = "{}: Release {} created".format(package.title, rel.title)
+				triggerNotif(package.author, current_user, msg, rel.getEditURL())
+				db.session.commit()
+
+		else:
+			flash("Unknown action: " + action, "error")
+
+	deleted_packages = Package.query.filter_by(soft_deleted=True).all()
+	return render_template("admin/list.html", deleted_packages=deleted_packages)
+
+class SwitchUserForm(FlaskForm):
+	username = StringField("Username")
+	submit = SubmitField("Switch")
+
+
+@bp.route("/admin/switchuser/", methods=["GET", "POST"])
+@rank_required(UserRank.ADMIN)
+def switch_user():
+	form = SwitchUserForm(formdata=request.form)
+	if request.method == "POST" and form.validate():
+		user = User.query.filter_by(username=form["username"].data).first()
+		if user is None:
+			flash("Unable to find user", "error")
+		elif loginUser(user):
+			return redirect(url_for("users.profile", username=current_user.username))
+		else:
+			flash("Unable to login as user", "error")
+
+
+	# Process GET or invalid POST
+	return render_template("admin/switch_user.html", form=form)

app/blueprints/admin/licenseseditor.py

@@ -0,0 +1,62 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import rank_required
+
+@bp.route("/licenses/")
+@rank_required(UserRank.MODERATOR)
+def license_list():
+	return render_template("admin/licenses/list.html", licenses=License.query.order_by(db.asc(License.name)).all())
+
+class LicenseForm(FlaskForm):
+	name	 = StringField("Name", [InputRequired(), Length(3,100)])
+	is_foss  = BooleanField("Is FOSS")
+	submit   = SubmitField("Save")
+
+@bp.route("/licenses/new/", methods=["GET", "POST"])
+@bp.route("/licenses/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def create_edit_license(name=None):
+	license = None
+	if name is not None:
+		license = License.query.filter_by(name=name).first()
+		if license is None:
+			abort(404)
+
+	form = LicenseForm(formdata=request.form, obj=license)
+	if request.method == "GET" and license is None:
+		form.is_foss.data = True
+	elif request.method == "POST" and form.validate():
+		if license is None:
+			license = License(form.name.data)
+			db.session.add(license)
+			flash("Created license " + form.name.data, "success")
+		else:
+			flash("Updated license " + form.name.data, "success")
+
+		form.populate_obj(license)
+		db.session.commit()
+		return redirect(url_for("admin.license_list"))
+
+	return render_template("admin/licenses/edit.html", license=license, form=form)

app/blueprints/admin/tagseditor.py

@@ -0,0 +1,57 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import rank_required
+
+@bp.route("/tags/")
+@rank_required(UserRank.MODERATOR)
+def tag_list():
+	return render_template("admin/tags/list.html", tags=Tag.query.order_by(db.asc(Tag.title)).all())
+
+class TagForm(FlaskForm):
+	title	 = StringField("Title", [InputRequired(), Length(3,100)])
+	name     = StringField("Name", [Optional(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
+	submit   = SubmitField("Save")
+
+@bp.route("/tags/new/", methods=["GET", "POST"])
+@bp.route("/tags/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def create_edit_tag(name=None):
+	tag = None
+	if name is not None:
+		tag = Tag.query.filter_by(name=name).first()
+		if tag is None:
+			abort(404)
+
+	form = TagForm(formdata=request.form, obj=tag)
+	if request.method == "POST" and form.validate():
+		if tag is None:
+			tag = Tag(form.title.data)
+			db.session.add(tag)
+		else:
+			form.populate_obj(tag)
+		db.session.commit()
+		return redirect(url_for("admin.create_edit_tag", name=tag.name))
+
+	return render_template("admin/tags/edit.html", tag=tag, form=form)

app/blueprints/admin/versioneditor.py

@@ -0,0 +1,60 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import rank_required
+
+@bp.route("/versions/")
+@rank_required(UserRank.MODERATOR)
+def version_list():
+	return render_template("admin/versions/list.html", versions=MinetestRelease.query.order_by(db.asc(MinetestRelease.id)).all())
+
+class VersionForm(FlaskForm):
+	name	 = StringField("Name", [InputRequired(), Length(3,100)])
+	protocol = IntegerField("Protocol")
+	submit   = SubmitField("Save")
+
+@bp.route("/versions/new/", methods=["GET", "POST"])
+@bp.route("/versions/<name>/edit/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def create_edit_version(name=None):
+	version = None
+	if name is not None:
+		version = MinetestRelease.query.filter_by(name=name).first()
+		if version is None:
+			abort(404)
+
+	form = VersionForm(formdata=request.form, obj=version)
+	if request.method == "POST" and form.validate():
+		if version is None:
+			version = MinetestRelease(form.name.data)
+			db.session.add(version)
+			flash("Created version " + form.name.data, "success")
+		else:
+			flash("Updated version " + form.name.data, "success")
+
+		form.populate_obj(version)
+		db.session.commit()
+		return redirect(url_for("admin.version_list"))
+
+	return render_template("admin/versions/edit.html", version=version, form=form)

app/blueprints/api/__init__.py

@@ -0,0 +1,21 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import Blueprint
+
+bp = Blueprint("api", __name__)
+
+from . import tokens, endpoints

app/blueprints/api/auth.py

@@ -0,0 +1,42 @@
+# Content DB
+# Copyright (C) 2019  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import request, make_response, jsonify, abort
+from app.models import APIToken
+from functools import wraps
+
+def is_api_authd(f):
+	@wraps(f)
+	def decorated_function(*args, **kwargs):
+		token = None
+
+		value = request.headers.get("authorization")
+		if value is None:
+			pass
+		elif value[0:7].lower() == "bearer ":
+			access_token = value[7:]
+			if len(access_token) < 10:
+				abort(400)
+
+			token = APIToken.query.filter_by(access_token=access_token).first()
+			if token is None:
+				abort(403)
+		else:
+			abort(403)
+
+		return f(token=token, *args, **kwargs)
+
+	return decorated_function

app/blueprints/api/endpoints.py

@@ -0,0 +1,109 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from . import bp
+from .auth import is_api_authd
+from app.models import *
+from app.utils import is_package_page
+from app.querybuilder import QueryBuilder
+
+@bp.route("/api/packages/")
+def packages():
+	qb    = QueryBuilder(request.args)
+	query = qb.buildPackageQuery()
+	ver   = qb.getMinetestVersion()
+
+	pkgs = [package.getAsDictionaryShort(current_app.config["BASE_URL"], version=ver) \
+			for package in query.all()]
+	return jsonify(pkgs)
+
+
+@bp.route("/api/packages/<author>/<name>/")
+@is_package_page
+def package(package):
+	return jsonify(package.getAsDictionary(current_app.config["BASE_URL"]))
+
+
+@bp.route("/api/packages/<author>/<name>/dependencies/")
+@is_package_page
+def package_dependencies(package):
+	ret = []
+
+	for dep in package.dependencies:
+		name = None
+		fulfilled_by = None
+
+		if dep.package:
+			name = dep.package.name
+			fulfilled_by = [ dep.package.getAsDictionaryKey() ]
+
+		elif dep.meta_package:
+			name = dep.meta_package.name
+			fulfilled_by = [ pkg.getAsDictionaryKey() for pkg in dep.meta_package.packages]
+
+		else:
+			raise "Malformed dependency"
+
+		ret.append({
+			"name": name,
+			"is_optional": dep.optional,
+			"packages": fulfilled_by
+		})
+
+	return jsonify(ret)
+
+
+@bp.route("/api/topics/")
+def topics():
+	qb     = QueryBuilder(request.args)
+	query  = qb.buildTopicQuery(show_added=True)
+	return jsonify([t.getAsDictionary() for t in query.all()])
+
+
+@bp.route("/api/topic_discard/", methods=["POST"])
+@login_required
+def topic_set_discard():
+	tid = request.args.get("tid")
+	discard = request.args.get("discard")
+	if tid is None or discard is None:
+		abort(400)
+
+	topic = ForumTopic.query.get(tid)
+	if not topic.checkPerm(current_user, Permission.TOPIC_DISCARD):
+		abort(403)
+
+	topic.discarded = discard == "true"
+	db.session.commit()
+
+	return jsonify(topic.getAsDictionary())
+
+
+@bp.route("/api/minetest_versions/")
+def versions():
+	return jsonify([{ "name": rel.name, "protocol_version": rel.protocol }\
+			for rel in MinetestRelease.query.all() if rel.getActual() is not None])
+
+
+@bp.route("/api/whoami/")
+@is_api_authd
+def whoami(token):
+	if token is None:
+		return jsonify({ "is_authenticated": False, "username": None })
+	else:
+		return jsonify({ "is_authenticated": True, "username": token.owner.username })

app/blueprints/api/tokens.py

@@ -0,0 +1,141 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import render_template, redirect, request, session, url_for, abort
+from flask_user import login_required, current_user
+from . import bp
+from app.models import db, User, APIToken, Package, Permission
+from app.utils import randomString
+from app.querybuilder import QueryBuilder
+
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField
+
+class CreateAPIToken(FlaskForm):
+	name	     = StringField("Name", [InputRequired(), Length(1, 30)])
+	submit	     = SubmitField("Save")
+
+
+@bp.route("/users/<username>/tokens/")
+@login_required
+def list_tokens(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	return render_template("api/list_tokens.html", user=user)
+
+
+@bp.route("/users/<username>/tokens/new/", methods=["GET", "POST"])
+@bp.route("/users/<username>/tokens/<int:id>/edit/", methods=["GET", "POST"])
+@login_required
+def create_edit_token(username, id=None):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = None
+	access_token = None
+	if not is_new:
+		token = APIToken.query.get(id)
+		if token is None:
+			abort(404)
+		elif token.owner != user:
+			abort(403)
+
+		access_token = session.pop("token_" + str(id), None)
+
+	form = CreateAPIToken(formdata=request.form, obj=token)
+	if request.method == "POST" and form.validate():
+		if is_new:
+			token = APIToken()
+			token.owner = user
+			token.access_token = randomString(32)
+
+		form.populate_obj(token)
+		db.session.add(token)
+
+		db.session.commit() # save
+
+		# Store token so it can be shown in the edit page
+		session["token_" + str(token.id)] = token.access_token
+
+		return redirect(url_for("api.create_edit_token", username=username, id=token.id))
+
+	return render_template("api/create_edit_token.html", user=user, form=form, token=token, access_token=access_token)
+
+
+@bp.route("/users/<username>/tokens/<int:id>/reset/", methods=["POST"])
+@login_required
+def reset_token(username, id):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = APIToken.query.get(id)
+	if token is None:
+		abort(404)
+	elif token.owner != user:
+		abort(403)
+
+	token.access_token = randomString(32)
+
+	db.session.commit() # save
+
+	# Store token so it can be shown in the edit page
+	session["token_" + str(token.id)] = token.access_token
+
+	return redirect(url_for("api.create_edit_token", username=username, id=token.id))
+
+
+@bp.route("/users/<username>/tokens/<int:id>/delete/", methods=["POST"])
+@login_required
+def delete_token(username, id):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if not user.checkPerm(current_user, Permission.CREATE_TOKEN):
+		abort(403)
+
+	is_new = id is None
+
+	token = APIToken.query.get(id)
+	if token is None:
+		abort(404)
+	elif token.owner != user:
+		abort(403)
+
+	db.session.delete(token)
+	db.session.commit()
+
+	return redirect(url_for("api.list_tokens", username=username))

app/blueprints/homepage/__init__.py

@@ -0,0 +1,21 @@
+from flask import Blueprint, render_template
+
+bp = Blueprint("homepage", __name__)
+
+from app.models import *
+import flask_menu as menu
+from sqlalchemy.sql.expression import func
+
+@bp.route("/")
+@menu.register_menu(bp, ".", "Home")
+def home():
+	query   = Package.query.filter_by(approved=True, soft_deleted=False)
+	count   = query.count()
+	new     = query.order_by(db.desc(Package.created_at)).limit(8).all()
+	pop_mod = query.filter_by(type=PackageType.MOD).order_by(db.desc(Package.score)).limit(8).all()
+	pop_gam = query.filter_by(type=PackageType.GAME).order_by(db.desc(Package.score)).limit(4).all()
+	pop_txp = query.filter_by(type=PackageType.TXP).order_by(db.desc(Package.score)).limit(4).all()
+	downloads_result = db.session.query(func.sum(PackageRelease.downloads)).one_or_none()
+	downloads = 0 if not downloads_result or not downloads_result[0] else downloads_result[0]
+	return render_template("index.html", count=count, downloads=downloads, \
+			new=new, pop_mod=pop_mod, pop_txp=pop_txp, pop_gam=pop_gam)

app/blueprints/metapackages/__init__.py

@@ -0,0 +1,36 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+
+bp = Blueprint("metapackages", __name__)
+
+from flask_user import *
+from app.models import *
+
+@bp.route("/metapackages/")
+def list_all():
+	mpackages = MetaPackage.query.order_by(db.asc(MetaPackage.name)).all()
+	return render_template("meta/list.html", mpackages=mpackages)
+
+@bp.route("/metapackages/<name>/")
+def view(name):
+	mpackage = MetaPackage.query.filter_by(name=name).first()
+	if mpackage is None:
+		abort(404)
+
+	return render_template("meta/view.html", mpackage=mpackage)

app/blueprints/notifications/__init__.py

@@ -0,0 +1,34 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import Blueprint, render_template, redirect, url_for
+from flask_user import current_user, login_required
+from app.models import db
+
+bp = Blueprint("notifications", __name__)
+
+@bp.route("/notifications/")
+@login_required
+def list_all():
+	return render_template("notifications/list.html")
+
+@bp.route("/notifications/clear/", methods=["POST"])
+@login_required
+def clear():
+	current_user.notifications.clear()
+	db.session.commit()
+	return redirect(url_for("notifications.list_all"))

app/blueprints/packages/__init__.py

@@ -0,0 +1,21 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import Blueprint
+
+bp = Blueprint("packages", __name__)
+
+from . import packages, screenshots, releases

app/blueprints/packages/editrequests.py

@@ -14,10 +14,8 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
-
 from flask import *
 from flask_user import *
-from flask.ext import menu
 from app import app
 from app.models import *
 
@@ -58,8 +56,13 @@ def create_edit_editrequest_page(package, id=None):
 		edited_package = Package(package)
 		erequest.applyAll(edited_package)
 
-
 	form = EditRequestForm(request.form, obj=edited_package)
+	if request.method == "GET":
+		deps = edited_package.dependencies
+		form.harddep_str.data  = ",".join([str(x) for x in deps if not x.optional])
+		form.softdep_str.data  = ",".join([str(x) for x in deps if     x.optional])
+		form.provides_str.data = MetaPackage.ListToSpec(edited_package.provides)
+
 	if request.method == "POST" and form.validate():
 		if erequest is None:
 			erequest = EditRequest()

app/blueprints/packages/packages.py

@@ -0,0 +1,372 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import render_template, abort, request, redirect, url_for, flash
+from flask_user import current_user
+import flask_menu as menu
+
+from . import bp
+
+from app.models import *
+from app.querybuilder import QueryBuilder
+from app.tasks.importtasks import importRepoScreenshot
+from app.utils import *
+
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField, QuerySelectMultipleField
+from sqlalchemy import or_
+
+
+@menu.register_menu(bp, ".mods", "Mods", order=11, endpoint_arguments_constructor=lambda: { 'type': 'mod' })
+@menu.register_menu(bp, ".games", "Games", order=12, endpoint_arguments_constructor=lambda: { 'type': 'game' })
+@menu.register_menu(bp, ".txp", "Texture Packs", order=13, endpoint_arguments_constructor=lambda: { 'type': 'txp' })
+@menu.register_menu(bp, ".random", "Random", order=14, endpoint_arguments_constructor=lambda: { 'random': '1', 'lucky': '1' })
+@bp.route("/packages/")
+def list_all():
+	qb    = QueryBuilder(request.args)
+	query = qb.buildPackageQuery()
+	title = qb.title
+
+	if qb.lucky:
+		package = query.first()
+		if package:
+			return redirect(package.getDetailsURL())
+
+		topic = qb.buildTopicQuery().first()
+		if qb.search and topic:
+			return redirect("https://forum.minetest.net/viewtopic.php?t=" + str(topic.topic_id))
+
+	page  = get_int_or_abort(request.args.get("page"), 1)
+	num   = min(40, get_int_or_abort(request.args.get("n"), 100))
+	query = query.paginate(page, num, True)
+
+	search = request.args.get("q")
+	type_name = request.args.get("type")
+
+	next_url = url_for("packages.list_all", type=type_name, q=search, page=query.next_num) \
+			if query.has_next else None
+	prev_url = url_for("packages.list_all", type=type_name, q=search, page=query.prev_num) \
+			if query.has_prev else None
+
+	topics = None
+	if qb.search and not query.has_next:
+		topics = qb.buildTopicQuery().all()
+
+	tags = Tag.query.all()
+	return render_template("packages/list.html", \
+			title=title, packages=query.items, topics=topics, \
+			query=search, tags=tags, type=type_name, \
+			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, packages_count=query.total)
+
+
+def getReleases(package):
+	if package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		return package.releases.limit(5)
+	else:
+		return package.releases.filter_by(approved=True).limit(5)
+
+
+@bp.route("/packages/<author>/<name>/")
+@is_package_page
+def view(package):
+	clearNotifications(package.getDetailsURL())
+
+	alternatives = None
+	if package.type == PackageType.MOD:
+		alternatives = Package.query \
+			.filter_by(name=package.name, type=PackageType.MOD, soft_deleted=False) \
+			.filter(Package.id != package.id) \
+			.order_by(db.desc(Package.score)) \
+			.all()
+
+
+	show_similar_topics = current_user == package.author or \
+			package.checkPerm(current_user, Permission.APPROVE_NEW)
+
+	similar_topics = None if not show_similar_topics else \
+			ForumTopic.query \
+				.filter_by(name=package.name) \
+				.filter(ForumTopic.topic_id != package.forums) \
+				.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
+				.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
+				.all()
+
+	releases = getReleases(package)
+	requests = [r for r in package.requests if r.status == 0]
+
+	review_thread = package.review_thread
+	if review_thread is not None and not review_thread.checkPerm(current_user, Permission.SEE_THREAD):
+		review_thread = None
+
+	topic_error = None
+	topic_error_lvl = "warning"
+	if not package.approved and package.forums is not None:
+		errors = []
+		if Package.query.filter_by(forums=package.forums, soft_deleted=False).count() > 1:
+			errors.append("<b>Error: Another package already uses this forum topic!</b>")
+			topic_error_lvl = "danger"
+
+		topic = ForumTopic.query.get(package.forums)
+		if topic is not None:
+			if topic.author != package.author:
+				errors.append("<b>Error: Forum topic author doesn't match package author.</b>")
+				topic_error_lvl = "danger"
+
+			if topic.wip:
+				errors.append("Warning: Forum topic is in WIP section, make sure package meets playability standards.")
+		elif package.type != PackageType.TXP:
+			errors.append("Warning: Forum topic not found. This may happen if the topic has only just been created.")
+
+		topic_error = "<br />".join(errors)
+
+
+	threads = Thread.query.filter_by(package_id=package.id)
+	if not current_user.is_authenticated:
+		threads = threads.filter_by(private=False)
+	elif not current_user.rank.atLeast(UserRank.EDITOR) and not current_user == package.author:
+		threads = threads.filter(or_(Thread.private == False, Thread.author == current_user))
+
+
+	return render_template("packages/view.html", \
+			package=package, releases=releases, requests=requests, \
+			alternatives=alternatives, similar_topics=similar_topics, \
+			review_thread=review_thread, topic_error=topic_error, topic_error_lvl=topic_error_lvl, \
+			threads=threads.all())
+
+
+@bp.route("/packages/<author>/<name>/download/")
+@is_package_page
+def download(package):
+	release = package.getDownloadRelease()
+
+	if release is None:
+		if "application/zip" in request.accept_mimetypes and \
+				not "text/html" in request.accept_mimetypes:
+			return "", 204
+		else:
+			flash("No download available.", "error")
+			return redirect(package.getDetailsURL())
+	else:
+		PackageRelease.query.filter_by(id=release.id).update({
+				"downloads": PackageRelease.downloads + 1
+			})
+		db.session.commit()
+
+		return redirect(release.url, code=302)
+
+
+class PackageForm(FlaskForm):
+	name          = StringField("Name (Technical)", [InputRequired(), Length(1, 20), Regexp("^[a-z0-9_]", 0, "Lower case letters (a-z), digits (0-9), and underscores (_) only")])
+	title         = StringField("Title (Human-readable)", [InputRequired(), Length(3, 50)])
+	short_desc     = StringField("Short Description (Plaintext)", [InputRequired(), Length(1,200)])
+	desc          = TextAreaField("Long Description (Markdown)", [Optional(), Length(0,10000)])
+	type          = SelectField("Type", [InputRequired()], choices=PackageType.choices(), coerce=PackageType.coerce, default=PackageType.MOD)
+	license       = QuerySelectField("License", [InputRequired()], query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	media_license = QuerySelectField("Media License", [InputRequired()], query_factory=lambda: License.query.order_by(db.asc(License.name)), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	provides_str  = StringField("Provides (mods included in package)", [Optional()])
+	tags          = QuerySelectMultipleField('Tags', query_factory=lambda: Tag.query.order_by(db.asc(Tag.name)), get_pk=lambda a: a.id, get_label=lambda a: a.title)
+	harddep_str   = StringField("Hard Dependencies", [Optional()])
+	softdep_str   = StringField("Soft Dependencies", [Optional()])
+	repo          = StringField("VCS Repository URL", [Optional(), URL()], filters = [lambda x: x or None])
+	website       = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
+	issueTracker  = StringField("Issue Tracker URL", [Optional(), URL()], filters = [lambda x: x or None])
+	forums	      = IntegerField("Forum Topic ID", [Optional(), NumberRange(0,999999)])
+	submit	      = SubmitField("Save")
+
+@bp.route("/packages/new/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/edit/", methods=["GET", "POST"])
+@login_required
+def create_edit(author=None, name=None):
+	package = None
+	form = None
+	if author is None:
+		form = PackageForm(formdata=request.form)
+		author = request.args.get("author")
+		if author is None or author == current_user.username:
+			author = current_user
+		else:
+			author = User.query.filter_by(username=author).first()
+			if author is None:
+				flash("Unable to find that user", "error")
+				return redirect(url_for("packages.create_edit"))
+
+			if not author.checkPerm(current_user, Permission.CHANGE_AUTHOR):
+				flash("Permission denied", "error")
+				return redirect(url_for("packages.create_edit"))
+
+	else:
+		package = getPackageByInfo(author, name)
+		if not package.checkPerm(current_user, Permission.EDIT_PACKAGE):
+			return redirect(package.getDetailsURL())
+
+		author = package.author
+
+		form = PackageForm(formdata=request.form, obj=package)
+
+	# Initial form class from post data and default data
+	if request.method == "GET":
+		if package is None:
+			form.name.data   = request.args.get("bname")
+			form.title.data  = request.args.get("title")
+			form.repo.data   = request.args.get("repo")
+			form.forums.data = request.args.get("forums")
+		else:
+			deps = package.dependencies
+			form.harddep_str.data  = ",".join([str(x) for x in deps if not x.optional])
+			form.softdep_str.data  = ",".join([str(x) for x in deps if     x.optional])
+			form.provides_str.data = MetaPackage.ListToSpec(package.provides)
+
+	if request.method == "POST" and form.validate():
+		wasNew = False
+		if not package:
+			package = Package.query.filter_by(name=form["name"].data, author_id=author.id).first()
+			if package is not None:
+				if package.soft_deleted:
+					Package.query.filter_by(name=form["name"].data, author_id=author.id).delete()
+				else:
+					flash("Package already exists!", "error")
+					return redirect(url_for("packages.create_edit"))
+
+			package = Package()
+			package.author = author
+			wasNew = True
+
+		elif package.approved and package.name != form.name.data and \
+				not package.checkPerm(current_user, Permission.CHANGE_NAME):
+			flash("Unable to change package name", "danger")
+			return redirect(url_for("packages.create_edit", author=author, name=name))
+
+		else:
+			triggerNotif(package.author, current_user,
+					"{} edited".format(package.title), package.getDetailsURL())
+
+		form.populate_obj(package) # copy to row
+
+		if package.type== PackageType.TXP:
+			package.license = package.media_license
+
+		mpackage_cache = {}
+		package.provides.clear()
+		mpackages = MetaPackage.SpecToList(form.provides_str.data, mpackage_cache)
+		for m in mpackages:
+			package.provides.append(m)
+
+		Dependency.query.filter_by(depender=package).delete()
+		deps = Dependency.SpecToList(package, form.harddep_str.data, mpackage_cache)
+		for dep in deps:
+			dep.optional = False
+			db.session.add(dep)
+
+		deps = Dependency.SpecToList(package, form.softdep_str.data, mpackage_cache)
+		for dep in deps:
+			dep.optional = True
+			db.session.add(dep)
+
+		if wasNew and package.type == PackageType.MOD and not package.name in mpackage_cache:
+			m = MetaPackage.GetOrCreate(package.name, mpackage_cache)
+			package.provides.append(m)
+
+		package.tags.clear()
+		for tag in form.tags.raw_data:
+			package.tags.append(Tag.query.get(tag))
+
+		db.session.commit() # save
+
+		next_url = package.getDetailsURL()
+		if wasNew and package.repo is not None:
+			task = importRepoScreenshot.delay(package.id)
+			next_url = url_for("tasks.check", id=task.id, r=next_url)
+
+		if wasNew and ("WTFPL" in package.license.name or "WTFPL" in package.media_license.name):
+			next_url = url_for("flatpage", path="help/wtfpl", r=next_url)
+
+		return redirect(next_url)
+
+	package_query = Package.query.filter_by(approved=True, soft_deleted=False)
+	if package is not None:
+		package_query = package_query.filter(Package.id != package.id)
+
+	enableWizard = name is None and request.method != "POST"
+	return render_template("packages/create_edit.html", package=package, \
+			form=form, author=author, enable_wizard=enableWizard, \
+			packages=package_query.all(), \
+			mpackages=MetaPackage.query.order_by(db.asc(MetaPackage.name)).all())
+
+@bp.route("/packages/<author>/<name>/approve/", methods=["POST"])
+@login_required
+@is_package_page
+def approve(package):
+	if not package.checkPerm(current_user, Permission.APPROVE_NEW):
+		flash("You don't have permission to do that.", "error")
+
+	elif package.approved:
+		flash("Package has already been approved", "error")
+
+	else:
+		package.approved = True
+
+		screenshots = PackageScreenshot.query.filter_by(package=package, approved=False).all()
+		for s in screenshots:
+			s.approved = True
+
+		triggerNotif(package.author, current_user,
+				"{} approved".format(package.title), package.getDetailsURL())
+		db.session.commit()
+
+	return redirect(package.getDetailsURL())
+
+
+@bp.route("/packages/<author>/<name>/remove/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def remove(package):
+	if request.method == "GET":
+		return render_template("packages/remove.html", package=package)
+
+	if "delete" in request.form:
+		if not package.checkPerm(current_user, Permission.DELETE_PACKAGE):
+			flash("You don't have permission to do that.", "error")
+			return redirect(package.getDetailsURL())
+
+		package.soft_deleted = True
+
+		url = url_for("users.profile", username=package.author.username)
+		triggerNotif(package.author, current_user,
+				"{} deleted".format(package.title), url)
+		db.session.commit()
+
+		flash("Deleted package", "success")
+
+		return redirect(url)
+	elif "unapprove" in request.form:
+		if not package.checkPerm(current_user, Permission.UNAPPROVE_PACKAGE):
+			flash("You don't have permission to do that.", "error")
+			return redirect(package.getDetailsURL())
+
+		package.approved = False
+
+		triggerNotif(package.author, current_user,
+				"{} deleted".format(package.title), package.getDetailsURL())
+		db.session.commit()
+
+		flash("Unapproved package", "success")
+
+		return redirect(package.getDetailsURL())
+	else:
+		abort(400)

app/blueprints/packages/releases.py

@@ -0,0 +1,252 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+
+from . import bp
+
+from app.rediscache import has_key, set_key, make_download_key
+from app.models import *
+from app.tasks.importtasks import makeVCSRelease, checkZipRelease
+from app.utils import *
+
+from celery import uuid
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from wtforms.ext.sqlalchemy.fields import QuerySelectField
+
+
+def get_mt_releases(is_max):
+	query = MinetestRelease.query.order_by(db.asc(MinetestRelease.id))
+	if is_max:
+		query = query.limit(query.count() - 1)
+	else:
+		query = query.filter(MinetestRelease.name != "0.4.17")
+
+	return query
+
+
+class CreatePackageReleaseForm(FlaskForm):
+	title	   = StringField("Title", [InputRequired(), Length(1, 30)])
+	uploadOpt  = RadioField ("Method", choices=[("upload", "File Upload")], default="upload")
+	vcsLabel   = StringField("VCS Commit Hash, Branch, or Tag", default="master")
+	fileUpload = FileField("File Upload")
+	min_rel    = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	max_rel    = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	submit	   = SubmitField("Save")
+
+class EditPackageReleaseForm(FlaskForm):
+	title    = StringField("Title", [InputRequired(), Length(1, 30)])
+	url      = StringField("URL", [URL])
+	task_id  = StringField("Task ID", filters = [lambda x: x or None])
+	approved = BooleanField("Is Approved")
+	min_rel  = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	max_rel  = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	submit   = SubmitField("Save")
+
+@bp.route("/packages/<author>/<name>/releases/new/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def create_release(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = CreatePackageReleaseForm()
+	if package.repo is not None:
+		form["uploadOpt"].choices = [("vcs", "From Git Commit or Branch"), ("upload", "File Upload")]
+		if request.method != "POST":
+			form["uploadOpt"].data = "vcs"
+
+	if request.method == "POST" and form.validate():
+		if form["uploadOpt"].data == "vcs":
+			rel = PackageRelease()
+			rel.package = package
+			rel.title   = form["title"].data
+			rel.url     = ""
+			rel.task_id = uuid()
+			rel.min_rel = form["min_rel"].data.getActual()
+			rel.max_rel = form["max_rel"].data.getActual()
+			db.session.add(rel)
+			db.session.commit()
+
+			makeVCSRelease.apply_async((rel.id, form["vcsLabel"].data), task_id=rel.task_id)
+
+			msg = "{}: Release {} created".format(package.title, rel.title)
+			triggerNotif(package.author, current_user, msg, rel.getEditURL())
+			db.session.commit()
+
+			return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
+		else:
+			uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "zip", "a zip file")
+			if uploadedUrl is not None:
+				rel = PackageRelease()
+				rel.package = package
+				rel.title = form["title"].data
+				rel.url = uploadedUrl
+				rel.task_id = uuid()
+				rel.min_rel = form["min_rel"].data.getActual()
+				rel.max_rel = form["max_rel"].data.getActual()
+				db.session.add(rel)
+				db.session.commit()
+
+				checkZipRelease.apply_async((rel.id, uploadedPath), task_id=rel.task_id)
+
+				msg = "{}: Release {} created".format(package.title, rel.title)
+				triggerNotif(package.author, current_user, msg, rel.getEditURL())
+				db.session.commit()
+
+				return redirect(url_for("tasks.check", id=rel.task_id, r=rel.getEditURL()))
+
+	return render_template("packages/release_new.html", package=package, form=form)
+
+@bp.route("/packages/<author>/<name>/releases/<id>/download/")
+@is_package_page
+def download_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	ip = request.headers.get("X-Forwarded-For") or request.remote_addr
+	if ip is not None:
+		key = make_download_key(ip, release.package)
+		if not has_key(key):
+			set_key(key, "true")
+
+			bonus = 1
+			if not package.getIsFOSS():
+				bonus *= 0.1
+
+			PackageRelease.query.filter_by(id=release.id).update({
+					"downloads": PackageRelease.downloads + 1
+				})
+
+			Package.query.filter_by(id=package.id).update({
+					"score": Package.score + bonus
+				})
+
+			db.session.commit()
+
+	return redirect(release.url, code=300)
+
+@bp.route("/packages/<author>/<name>/releases/<id>/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def edit_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	clearNotifications(release.getEditURL())
+
+	canEdit	= package.checkPerm(current_user, Permission.MAKE_RELEASE)
+	canApprove = package.checkPerm(current_user, Permission.APPROVE_RELEASE)
+	if not (canEdit or canApprove):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = EditPackageReleaseForm(formdata=request.form, obj=release)
+
+	# HACK: fix bug in wtforms
+	if request.method == "GET":
+		form.approved.data = release.approved
+
+	if request.method == "POST" and form.validate():
+		wasApproved = release.approved
+		if canEdit:
+			release.title = form["title"].data
+			release.min_rel = form["min_rel"].data.getActual()
+			release.max_rel = form["max_rel"].data.getActual()
+
+		if package.checkPerm(current_user, Permission.CHANGE_RELEASE_URL):
+			release.url = form["url"].data
+			release.task_id = form["task_id"].data
+			if release.task_id is not None:
+				release.task_id = None
+
+		if canApprove:
+			release.approved = form["approved"].data
+		else:
+			release.approved = wasApproved
+
+		db.session.commit()
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/release_edit.html", package=package, release=release, form=form)
+
+
+
+class BulkReleaseForm(FlaskForm):
+	set_min = BooleanField("Set Min")
+	min_rel  = QuerySelectField("Minimum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(False), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	set_max = BooleanField("Set Max")
+	max_rel  = QuerySelectField("Maximum Minetest Version", [InputRequired()],
+			query_factory=lambda: get_mt_releases(True), get_pk=lambda a: a.id, get_label=lambda a: a.name)
+	only_change_none = BooleanField("Only change values previously set as none")
+	submit   = SubmitField("Update")
+
+
+@bp.route("/packages/<author>/<name>/releases/bulk_change/", methods=["GET", "POST"])
+@login_required
+@is_package_page
+def bulk_change_release(package):
+	if not package.checkPerm(current_user, Permission.MAKE_RELEASE):
+		return redirect(package.getDetailsURL())
+
+	# Initial form class from post data and default data
+	form = BulkReleaseForm()
+
+	if request.method == "GET":
+		form.only_change_none.data = True
+	elif request.method == "POST" and form.validate():
+		only_change_none = form.only_change_none.data
+
+		for release in package.releases.all():
+			if form["set_min"].data and (not only_change_none or release.min_rel is None):
+				release.min_rel = form["min_rel"].data.getActual()
+			if form["set_max"].data and (not only_change_none or release.max_rel is None):
+				release.max_rel = form["max_rel"].data.getActual()
+
+		db.session.commit()
+
+		return redirect(package.getDetailsURL())
+
+	return render_template("packages/release_bulk_change.html", package=package, form=form)
+
+
+@bp.route("/packages/<author>/<name>/releases/<id>/delete/", methods=["POST"])
+@login_required
+@is_package_page
+def delete_release(package, id):
+	release = PackageRelease.query.get(id)
+	if release is None or release.package != package:
+		abort(404)
+
+	if not release.checkPerm(current_user, Permission.DELETE_RELEASE):
+		return redirect(release.getEditURL())
+
+	db.session.delete(release)
+	db.session.commit()
+
+	return redirect(package.getDetailsURL())

app/blueprints/packages/screenshots.py

@@ -17,9 +17,10 @@
 
 from flask import *
 from flask_user import *
-from app import app
-from app.models import *
 
+from . import bp
+
+from app.models import *
 from app.utils import *
 
 from flask_wtf import FlaskForm
@@ -39,23 +40,24 @@ class EditScreenshotForm(FlaskForm):
 	delete   = BooleanField("Delete")
 	submit   = SubmitField("Save")
 
-@app.route("/packages/<author>/<name>/screenshots/new/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/screenshots/new/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def create_screenshot_page(package, id=None):
+def create_screenshot(package, id=None):
 	if not package.checkPerm(current_user, Permission.ADD_SCREENSHOTS):
 		return redirect(package.getDetailsURL())
 
 	# Initial form class from post data and default data
 	form = CreateScreenshotForm()
 	if request.method == "POST" and form.validate():
-		uploadedPath = doFileUpload(form.fileUpload.data, ["png", "jpg", "jpeg"],
+		uploadedUrl, uploadedPath = doFileUpload(form.fileUpload.data, "image",
 				"a PNG or JPG image file")
-		if uploadedPath is not None:
+		if uploadedUrl is not None:
 			ss = PackageScreenshot()
-			ss.package = package
-			ss.title   = form["title"].data
-			ss.url     = uploadedPath
+			ss.package  = package
+			ss.title    = form["title"].data or "Untitled"
+			ss.url      = uploadedUrl
+			ss.approved = package.checkPerm(current_user, Permission.APPROVE_SCREENSHOT)
 			db.session.add(ss)
 
 			msg = "{}: Screenshot added {}" \
@@ -66,10 +68,10 @@ def create_screenshot_page(package, id=None):
 
 	return render_template("packages/screenshot_new.html", package=package, form=form)
 
-@app.route("/packages/<author>/<name>/screenshots/<id>/edit/", methods=["GET", "POST"])
+@bp.route("/packages/<author>/<name>/screenshots/<id>/edit/", methods=["GET", "POST"])
 @login_required
 @is_package_page
-def edit_screenshot_page(package, id):
+def edit_screenshot(package, id):
 	screenshot = PackageScreenshot.query.get(id)
 	if screenshot is None or screenshot.package != package:
 		abort(404)
@@ -91,7 +93,7 @@ def edit_screenshot_page(package, id):
 			wasApproved = screenshot.approved
 
 			if canEdit:
-				screenshot.title = form["title"].data
+				screenshot.title = form["title"].data or "Untitled"
 
 			if canApprove:
 				screenshot.approved = form["approved"].data

app/blueprints/tasks/__init__.py

@@ -17,29 +17,29 @@
 
 from flask import *
 from flask_user import *
-from flask.ext import menu
-from app import app, csrf
+import flask_menu as menu
+from app import csrf
 from app.models import *
 from app.tasks import celery, TaskError
 from app.tasks.importtasks import getMeta
 from app.utils import shouldReturnJson
-# from celery.result import AsyncResult
-
 from app.utils import *
 
+bp = Blueprint("tasks", __name__)
+
 @csrf.exempt
-@app.route("/tasks/getmeta/new/", methods=["POST"])
+@bp.route("/tasks/getmeta/new/", methods=["POST"])
 @login_required
-def new_getmeta_page():
+def start_getmeta():
 	author = request.args.get("author")
 	author = current_user.forums_username if author is None else author
 	aresult = getMeta.delay(request.args.get("url"), author)
 	return jsonify({
-		"poll_url": url_for("check_task", id=aresult.id),
+		"poll_url": url_for("tasks.check", id=aresult.id),
 	})
 
-@app.route("/tasks/<id>/")
-def check_task(id):
+@bp.route("/tasks/<id>/")
+def check(id):
 	result = celery.AsyncResult(id)
 	status = result.status
 	traceback = result.traceback

app/blueprints/threads/__init__.py

@@ -0,0 +1,214 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+
+bp = Blueprint("threads", __name__)
+
+from flask_user import *
+from app.models import *
+from app.utils import triggerNotif, clearNotifications
+
+import datetime
+
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+
+@bp.route("/threads/")
+def list_all():
+	query = Thread.query
+	if not Permission.SEE_THREAD.check(current_user):
+		query = query.filter_by(private=False)
+	return render_template("threads/list.html", threads=query.all())
+
+
+@bp.route("/threads/<int:id>/subscribe/", methods=["POST"])
+@login_required
+def subscribe(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
+		abort(404)
+
+	if current_user in thread.watchers:
+		flash("Already subscribed!", "success")
+	else:
+		flash("Subscribed to thread", "success")
+		thread.watchers.append(current_user)
+		db.session.commit()
+
+	return redirect(url_for("threads.view", id=id))
+
+
+@bp.route("/threads/<int:id>/unsubscribe/", methods=["POST"])
+@login_required
+def unsubscribe(id):
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
+		abort(404)
+
+	if current_user in thread.watchers:
+		flash("Unsubscribed!", "success")
+		thread.watchers.remove(current_user)
+		db.session.commit()
+	else:
+		flash("Not subscribed to thread", "success")
+
+	return redirect(url_for("threads.view", id=id))
+
+
+@bp.route("/threads/<int:id>/", methods=["GET", "POST"])
+def view(id):
+	clearNotifications(url_for("threads.view", id=id))
+
+	thread = Thread.query.get(id)
+	if thread is None or not thread.checkPerm(current_user, Permission.SEE_THREAD):
+		abort(404)
+
+	if current_user.is_authenticated and request.method == "POST":
+		comment = request.form["comment"]
+
+		if not current_user.canCommentRL():
+			flash("Please wait before commenting again", "danger")
+			if package:
+				return redirect(package.getDetailsURL())
+			else:
+				return redirect(url_for("homepage.home"))
+
+		if len(comment) <= 500 and len(comment) > 3:
+			reply = ThreadReply()
+			reply.author = current_user
+			reply.comment = comment
+			db.session.add(reply)
+
+			thread.replies.append(reply)
+			if not current_user in thread.watchers:
+				thread.watchers.append(current_user)
+
+			msg = None
+			if thread.package is None:
+				msg = "New comment on '{}'".format(thread.title)
+			else:
+				msg = "New comment on '{}' on package {}".format(thread.title, thread.package.title)
+
+
+			for user in thread.watchers:
+				if user != current_user:
+					triggerNotif(user, current_user, msg, url_for("threads.view", id=thread.id))
+
+			db.session.commit()
+
+			return redirect(url_for("threads.view", id=id))
+
+		else:
+			flash("Comment needs to be between 3 and 500 characters.")
+
+	return render_template("threads/view.html", thread=thread)
+
+
+class ThreadForm(FlaskForm):
+	title	= StringField("Title", [InputRequired(), Length(3,100)])
+	comment = TextAreaField("Comment", [InputRequired(), Length(10, 500)])
+	private = BooleanField("Private")
+	submit  = SubmitField("Open Thread")
+
+@bp.route("/threads/new/", methods=["GET", "POST"])
+@login_required
+def new():
+	form = ThreadForm(formdata=request.form)
+
+	package = None
+	if "pid" in request.args:
+		package = Package.query.get(int(request.args.get("pid")))
+		if package is None:
+			flash("Unable to find that package!", "error")
+
+	# Don't allow making orphan threads on approved packages for now
+	if package is None:
+		abort(403)
+
+	def_is_private   = request.args.get("private") or False
+	if package is None:
+		def_is_private = True
+	allow_change     = package and package.approved
+	is_review_thread = package and not package.approved
+
+	# Check that user can make the thread
+	if not package.checkPerm(current_user, Permission.CREATE_THREAD):
+		flash("Unable to create thread!", "error")
+		return redirect(url_for("homepage.home"))
+
+	# Only allow creating one thread when not approved
+	elif is_review_thread and package.review_thread is not None:
+		flash("A review thread already exists!", "error")
+		return redirect(url_for("threads.view", id=package.review_thread.id))
+
+	elif not current_user.canOpenThreadRL():
+		flash("Please wait before opening another thread", "danger")
+
+		if package:
+			return redirect(package.getDetailsURL())
+		else:
+			return redirect(url_for("homepage.home"))
+
+	# Set default values
+	elif request.method == "GET":
+		form.private.data = def_is_private
+		form.title.data   = request.args.get("title") or ""
+
+	# Validate and submit
+	elif request.method == "POST" and form.validate():
+		thread = Thread()
+		thread.author  = current_user
+		thread.title   = form.title.data
+		thread.private = form.private.data if allow_change else def_is_private
+		thread.package = package
+		db.session.add(thread)
+
+		thread.watchers.append(current_user)
+		if package is not None and package.author != current_user:
+			thread.watchers.append(package.author)
+
+		reply = ThreadReply()
+		reply.thread  = thread
+		reply.author  = current_user
+		reply.comment = form.comment.data
+		db.session.add(reply)
+
+		thread.replies.append(reply)
+
+		db.session.commit()
+
+		if is_review_thread:
+			package.review_thread = thread
+
+		notif_msg = None
+		if package is not None:
+			notif_msg = "New thread '{}' on package {}".format(thread.title, package.title)
+			triggerNotif(package.author, current_user, notif_msg, url_for("threads.view", id=thread.id))
+		else:
+			notif_msg = "New thread '{}'".format(thread.title)
+
+		for user in User.query.filter(User.rank >= UserRank.EDITOR).all():
+			triggerNotif(user, current_user, notif_msg, url_for("threads.view", id=thread.id))
+
+		db.session.commit()
+
+		return redirect(url_for("threads.view", id=thread.id))
+
+
+	return render_template("threads/new.html", form=form, allow_private_change=allow_change, package=package)

app/blueprints/thumbnails/__init__.py

@@ -0,0 +1,79 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+
+bp = Blueprint("thumbnails", __name__)
+
+import os
+from PIL import Image
+
+ALLOWED_RESOLUTIONS=[(100,67), (270,180), (350,233)]
+
+def mkdir(path):
+	assert path != "" and path is not None
+	if not os.path.isdir(path):
+		os.mkdir(path)
+
+
+def resize_and_crop(img_path, modified_path, size):
+	img = Image.open(img_path)
+
+	# Get current and desired ratio for the images
+	img_ratio = img.size[0] / float(img.size[1])
+	ratio = size[0] / float(size[1])
+
+	# Is more portrait than target, scale and crop
+	if ratio > img_ratio:
+		img = img.resize((int(size[0]), int(size[0] * img.size[1] / img.size[0])),
+				Image.BICUBIC)
+		box = (0, (img.size[1] - size[1]) / 2, img.size[0], (img.size[1] + size[1]) / 2)
+		img = img.crop(box)
+
+	# Is more landscape than target, scale and crop
+	elif ratio < img_ratio:
+		img = img.resize((int(size[1] * img.size[0] / img.size[1]), int(size[1])),
+				Image.BICUBIC)
+		box = ((img.size[0] - size[0]) / 2, 0, (img.size[0] + size[0]) / 2, img.size[1])
+		img = img.crop(box)
+
+	# Is exactly the same ratio as target
+	else:
+		img = img.resize(size, Image.BICUBIC)
+
+	img.save(modified_path)
+
+
+@bp.route("/thumbnails/<int:level>/<img>")
+def make_thumbnail(img, level):
+	if level > len(ALLOWED_RESOLUTIONS) or level <= 0:
+		abort(403)
+
+	w, h = ALLOWED_RESOLUTIONS[level - 1]
+
+	upload_dir = current_app.config["UPLOAD_DIR"]
+	thumbnail_dir = current_app.config["THUMBNAIL_DIR"]
+	mkdir(thumbnail_dir)
+
+	output_dir = os.path.join(thumbnail_dir, str(level))
+	mkdir(output_dir)
+
+	cache_filepath  = os.path.join(output_dir, img)
+	source_filepath = os.path.join(upload_dir, img)
+
+	resize_and_crop(source_filepath, cache_filepath, (w, h))
+	return send_file(cache_filepath)

app/blueprints/todo/__init__.py

@@ -0,0 +1,102 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+from flask import *
+from flask_user import *
+import flask_menu as menu
+from app.models import *
+from app.querybuilder import QueryBuilder
+from app.utils import get_int_or_abort
+
+bp = Blueprint("todo", __name__)
+
+@bp.route("/todo/", methods=["GET", "POST"])
+@login_required
+def view():
+	canApproveNew = Permission.APPROVE_NEW.check(current_user)
+	canApproveRel = Permission.APPROVE_RELEASE.check(current_user)
+	canApproveScn = Permission.APPROVE_SCREENSHOT.check(current_user)
+
+	packages = None
+	if canApproveNew:
+		packages = Package.query.filter_by(approved=False, soft_deleted=False).order_by(db.desc(Package.created_at)).all()
+
+	releases = None
+	if canApproveRel:
+		releases = PackageRelease.query.filter_by(approved=False).all()
+
+	screenshots = None
+	if canApproveScn:
+		screenshots = PackageScreenshot.query.filter_by(approved=False).all()
+
+	if not canApproveNew and not canApproveRel and not canApproveScn:
+		abort(403)
+
+	if request.method == "POST":
+		if request.form["action"] == "screenshots_approve_all":
+			if not canApproveScn:
+				abort(403)
+
+			PackageScreenshot.query.update({ "approved": True })
+			db.session.commit()
+			return redirect(url_for("todo.view"))
+		else:
+			abort(400)
+
+	topic_query = ForumTopic.query \
+			.filter_by(discarded=False)
+
+	total_topics = topic_query.count()
+	topics_to_add = topic_query \
+			.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
+			.count()
+
+	return render_template("todo/list.html", title="Reports and Work Queue",
+		packages=packages, releases=releases, screenshots=screenshots,
+		canApproveNew=canApproveNew, canApproveRel=canApproveRel, canApproveScn=canApproveScn,
+		topics_to_add=topics_to_add, total_topics=total_topics)
+
+
+@bp.route("/todo/topics/")
+@login_required
+def topics():
+	qb    = QueryBuilder(request.args)
+	qb.setSortIfNone("date")
+	query = qb.buildTopicQuery()
+
+	tmp_q = ForumTopic.query
+	if not qb.show_discarded:
+		tmp_q = tmp_q.filter_by(discarded=False)
+	total = tmp_q.count()
+	topic_count = query.count()
+
+	page  = get_int_or_abort(request.args.get("page"), 1)
+	num   = get_int_or_abort(request.args.get("n"), 100)
+	if num > 100 and not current_user.rank.atLeast(UserRank.EDITOR):
+		num = 100
+
+	query = query.paginate(page, num, True)
+	next_url = url_for("todo.topics", page=query.next_num, query=qb.search, \
+	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+			if query.has_next else None
+	prev_url = url_for("todo.topics", page=query.prev_num, query=qb.search, \
+	 	show_discarded=qb.show_discarded, n=num, sort=qb.order_by) \
+			if query.has_prev else None
+
+	return render_template("todo/topics.html", topics=query.items, total=total, \
+			topic_count=topic_count, query=qb.search, show_discarded=qb.show_discarded, \
+			next_url=next_url, prev_url=prev_url, page=page, page_max=query.pages, \
+			n=num, sort_by=qb.order_by)

app/blueprints/users/__init__.py

@@ -0,0 +1,5 @@
+from flask import Blueprint
+
+bp = Blueprint("users", __name__)
+
+from . import githublogin, profile

app/blueprints/users/githublogin.py

@@ -21,15 +21,16 @@ from flask_login import login_user, logout_user
 from sqlalchemy import func
 import flask_menu as menu
 from flask_github import GitHub
-from app import app, github
+from . import bp
+from app import github
 from app.models import *
 from app.utils import loginUser
 
-@app.route("/user/github/start/")
-def github_signin_page():
+@bp.route("/user/github/start/")
+def github_signin():
 	return github.authorize("")
 
-@app.route("/user/github/callback/")
+@bp.route("/user/github/callback/")
 @github.authorized_handler
 def github_authorized(oauth_token):
 	next_url = request.args.get("next")
@@ -51,20 +52,23 @@ def github_authorized(oauth_token):
 	if current_user and current_user.is_authenticated:
 		if userByGithub is None:
 			current_user.github_username = username
-			db.session.add(auth)
 			db.session.commit()
-			return redirect(url_for("gitAccount", id=auth.id))
+			flash("Linked github to account", "success")
+			return redirect(url_for("homepage.home"))
 		else:
 			flash("Github account is already associated with another user", "danger")
-			return redirect(url_for("home_page"))
+			return redirect(url_for("homepage.home"))
 
 	# If not logged in, log in
 	else:
 		if userByGithub is None:
 			flash("Unable to find an account for that Github user", "error")
-			return redirect(url_for("user_claim_page"))
+			return redirect(url_for("users.claim"))
 		elif loginUser(userByGithub):
-			return redirect(next_url or url_for("home_page"))
+			if not current_user.hasPassword():
+				return redirect(next_url or url_for("users.set_password", optional=True))
+			else:
+				return redirect(next_url or url_for("homepage.home"))
 		else:
 			flash("Authorization failed [err=gh-login-failed]", "danger")
 			return redirect(url_for("user.login"))

app/blueprints/users/profile.py

@@ -0,0 +1,310 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from flask import *
+from flask_user import *
+from flask_login import login_user, logout_user
+from app import markdown
+from . import bp
+from app.models import *
+from flask_wtf import FlaskForm
+from wtforms import *
+from wtforms.validators import *
+from app.utils import randomString, loginUser, rank_required
+from app.tasks.forumtasks import checkForumAccount
+from app.tasks.emails import sendVerifyEmail, sendEmailRaw
+from app.tasks.phpbbparser import getProfile
+
+# Define the User profile form
+class UserProfileForm(FlaskForm):
+	display_name = StringField("Display name", [Optional(), Length(2, 20)])
+	email = StringField("Email", [Optional(), Email()], filters = [lambda x: x or None])
+	website_url = StringField("Website URL", [Optional(), URL()], filters = [lambda x: x or None])
+	donate_url = StringField("Donation URL", [Optional(), URL()], filters = [lambda x: x or None])
+	rank = SelectField("Rank", [Optional()], choices=UserRank.choices(), coerce=UserRank.coerce, default=UserRank.NEW_MEMBER)
+	submit = SubmitField("Save")
+
+
+@bp.route("/users/", methods=["GET"])
+def list_all():
+	users = User.query.order_by(db.desc(User.rank), db.asc(User.display_name)).all()
+	return render_template("users/list.html", users=users)
+
+
+@bp.route("/users/<username>/", methods=["GET", "POST"])
+def profile(username):
+	user = User.query.filter_by(username=username).first()
+	if not user:
+		abort(404)
+
+	form = None
+	if user.checkPerm(current_user, Permission.CHANGE_DNAME) or \
+			user.checkPerm(current_user, Permission.CHANGE_EMAIL) or \
+			user.checkPerm(current_user, Permission.CHANGE_RANK):
+		# Initialize form
+		form = UserProfileForm(formdata=request.form, obj=user)
+
+		# Process valid POST
+		if request.method=="POST" and form.validate():
+			# Copy form fields to user_profile fields
+			if user.checkPerm(current_user, Permission.CHANGE_DNAME):
+				user.display_name = form["display_name"].data
+				user.website_url  = form["website_url"].data
+				user.donate_url   = form["donate_url"].data
+
+			if user.checkPerm(current_user, Permission.CHANGE_RANK):
+				newRank = form["rank"].data
+				if current_user.rank.atLeast(newRank):
+					user.rank = form["rank"].data
+				else:
+					flash("Can't promote a user to a rank higher than yourself!", "error")
+
+			if user.checkPerm(current_user, Permission.CHANGE_EMAIL):
+				newEmail = form["email"].data
+				if newEmail != user.email and newEmail.strip() != "":
+					token = randomString(32)
+
+					ver = UserEmailVerification()
+					ver.user  = user
+					ver.token = token
+					ver.email = newEmail
+					db.session.add(ver)
+					db.session.commit()
+
+					task = sendVerifyEmail.delay(newEmail, token)
+					return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=username)))
+
+			# Save user_profile
+			db.session.commit()
+
+			# Redirect to home page
+			return redirect(url_for("users.profile", username=username))
+
+	packages = user.packages.filter_by(soft_deleted=False)
+	if not current_user.is_authenticated or (user != current_user and not current_user.canAccessTodoList()):
+		packages = packages.filter_by(approved=True)
+	packages = packages.order_by(db.asc(Package.title))
+
+	topics_to_add = None
+	if current_user == user or user.checkPerm(current_user, Permission.CHANGE_AUTHOR):
+		topics_to_add = ForumTopic.query \
+					.filter_by(author_id=user.id) \
+					.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id)) \
+					.order_by(db.asc(ForumTopic.name), db.asc(ForumTopic.title)) \
+					.all()
+
+	# Process GET or invalid POST
+	return render_template("users/profile.html",
+			user=user, form=form, packages=packages, topics_to_add=topics_to_add)
+
+
+@bp.route("/users/<username>/check/", methods=["POST"])
+@login_required
+def user_check(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	if current_user != user and not current_user.rank.atLeast(UserRank.MODERATOR):
+		abort(403)
+
+	if user.forums_username is None:
+		abort(404)
+
+	task = checkForumAccount.delay(user.forums_username)
+	next_url = url_for("users.profile", username=username)
+
+	return redirect(url_for("tasks.check", id=task.id, r=next_url))
+
+
+class SendEmailForm(FlaskForm):
+	subject = StringField("Subject", [InputRequired(), Length(1, 300)])
+	text    = TextAreaField("Message", [InputRequired()])
+	submit  = SubmitField("Send")
+
+
+@bp.route("/users/<username>/email/", methods=["GET", "POST"])
+@rank_required(UserRank.MODERATOR)
+def send_email(username):
+	user = User.query.filter_by(username=username).first()
+	if user is None:
+		abort(404)
+
+	next_url = url_for("users.profile", username=user.username)
+
+	if user.email is None:
+		flash("User has no email address!", "error")
+		return redirect(next_url)
+
+	form = SendEmailForm(request.form)
+	if form.validate_on_submit():
+		text = form.text.data
+		html = markdown(text)
+		task = sendEmailRaw.delay([user.email], form.subject.data, text, html)
+		return redirect(url_for("tasks.check", id=task.id, r=next_url))
+
+	return render_template("users/send_email.html", form=form)
+
+
+
+class SetPasswordForm(FlaskForm):
+	email = StringField("Email", [Optional(), Email()])
+	password = PasswordField("New password", [InputRequired(), Length(2, 100)])
+	password2 = PasswordField("Verify password", [InputRequired(), Length(2, 100)])
+	submit = SubmitField("Save")
+
+@bp.route("/user/set-password/", methods=["GET", "POST"])
+@login_required
+def set_password():
+	if current_user.hasPassword():
+		return redirect(url_for("user.change_password"))
+
+	form = SetPasswordForm(request.form)
+	if current_user.email == None:
+		form.email.validators = [InputRequired(), Email()]
+
+	if request.method == "POST" and form.validate():
+		one = form.password.data
+		two = form.password2.data
+		if one == two:
+			# Hash password
+			hashed_password = user_manager.hash_password(form.password.data)
+
+			# Change password
+			current_user.password = hashed_password
+			db.session.commit()
+
+			# Send 'password_changed' email
+			if user_manager.USER_ENABLE_EMAIL and current_user.email:
+				emails.send_password_changed_email(current_user)
+
+			# Send password_changed signal
+			signals.user_changed_password.send(current_app._get_current_object(), user=current_user)
+
+			# Prepare one-time system message
+			flash('Your password has been changed successfully.', 'success')
+
+			newEmail = form["email"].data
+			if newEmail != current_user.email and newEmail.strip() != "":
+				token = randomString(32)
+
+				ver = UserEmailVerification()
+				ver.user = current_user
+				ver.token = token
+				ver.email = newEmail
+				db.session.add(ver)
+				db.session.commit()
+
+				task = sendVerifyEmail.delay(newEmail, token)
+				return redirect(url_for("tasks.check", id=task.id, r=url_for("users.profile", username=current_user.username)))
+			else:
+				return redirect(url_for("user.login"))
+		else:
+			flash("Passwords do not match", "error")
+
+	return render_template("users/set_password.html", form=form, optional=request.args.get("optional"))
+
+
+@bp.route("/user/claim/", methods=["GET", "POST"])
+def claim():
+	username = request.args.get("username")
+	if username is None:
+		username = ""
+	else:
+		method = request.args.get("method")
+		user = User.query.filter_by(forums_username=username).first()
+		if user and user.rank.atLeast(UserRank.NEW_MEMBER):
+			flash("User has already been claimed", "error")
+			return redirect(url_for("users.claim"))
+		elif user is None and method == "github":
+			flash("Unable to get Github username for user", "error")
+			return redirect(url_for("users.claim"))
+		elif user is None:
+			flash("Unable to find that user", "error")
+			return redirect(url_for("users.claim"))
+
+		if user is not None and method == "github":
+			return redirect(url_for("users.github_signin"))
+
+	token = None
+	if "forum_token" in session:
+		token = session["forum_token"]
+	else:
+		token = randomString(32)
+		session["forum_token"] = token
+
+	if request.method == "POST":
+		ctype	= request.form.get("claim_type")
+		username = request.form.get("username")
+
+		if username is None or len(username.strip()) < 2:
+			flash("Invalid username", "error")
+		elif ctype == "github":
+			task = checkForumAccount.delay(username)
+			return redirect(url_for("tasks.check", id=task.id, r=url_for("users.claim", username=username, method="github")))
+		elif ctype == "forum":
+			user = User.query.filter_by(forums_username=username).first()
+			if user is not None and user.rank.atLeast(UserRank.NEW_MEMBER):
+				flash("That user has already been claimed!", "error")
+				return redirect(url_for("users.claim"))
+
+			# Get signature
+			sig = None
+			try:
+				profile = getProfile("https://forum.minetest.net", username)
+				sig = profile.signature
+			except IOError:
+				flash("Unable to get forum signature - does the user exist?", "error")
+				return redirect(url_for("users.claim", username=username))
+
+			# Look for key
+			if token in sig:
+				if user is None:
+					user = User(username)
+					user.forums_username = username
+					db.session.add(user)
+					db.session.commit()
+
+				if loginUser(user):
+					return redirect(url_for("users.set_password"))
+				else:
+					flash("Unable to login as user", "error")
+					return redirect(url_for("users.claim", username=username))
+
+			else:
+				flash("Could not find the key in your signature!", "error")
+				return redirect(url_for("users.claim", username=username))
+		else:
+			flash("Unknown claim type", "error")
+
+	return render_template("users/claim.html", username=username, key=token)
+
+@bp.route("/users/verify/")
+def verify_email():
+	token = request.args.get("token")
+	ver = UserEmailVerification.query.filter_by(token=token).first()
+	if ver is None:
+		flash("Unknown verification token!", "error")
+	else:
+		ver.user.email = ver.email
+		db.session.delete(ver)
+		db.session.commit()
+
+	if current_user.is_authenticated:
+		return redirect(url_for("users.profile", username=current_user.username))
+	else:
+		return redirect(url_for("homepage.home"))

app/default_data.py

@@ -1,42 +1,65 @@
-# Content DB
-# Copyright (C) 2018  rubenwardy
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+from .models import *
+from .utils import make_flask_user_password
 
 
-import os, sys, datetime
+def populate(session):
+	admin_user = User("rubenwardy")
+	admin_user.active = True
+	admin_user.password = make_flask_user_password("tuckfrump")
+	admin_user.github_username = "rubenwardy"
+	admin_user.forums_username = "rubenwardy"
+	admin_user.rank = UserRank.ADMIN
+	session.add(admin_user)
 
-if not "FLASK_CONFIG" in os.environ:
-	os.environ["FLASK_CONFIG"] = "../config.cfg"
+	session.add(MinetestRelease("None", 0))
+	session.add(MinetestRelease("0.4.16/17", 32))
+	session.add(MinetestRelease("5.0", 37))
+	session.add(MinetestRelease("5.1", 38))
 
-test_data = len(sys.argv) >= 2 and sys.argv[1].strip() == "-t"
+	tags = {}
+	for tag in ["Inventory", "Mapgen", "Building", \
+			"Mobs and NPCs", "Tools", "Player effects", \
+			"Environment", "Transport", "Maintenance", "Plants and farming", \
+			"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
+		row = Tag(tag)
+		tags[row.name] = row
+		session.add(row)
 
-from app.models import *
+	licenses = {}
+	for license in ["GPLv2.1", "GPLv3", "LGPLv2.1", "LGPLv3", "AGPLv2.1", "AGPLv3",
+					"Apache", "BSD 3-Clause", "BSD 2-Clause", "CC0", "CC-BY-SA",
+					"CC-BY", "MIT", "ZLib", "Other (Free)"]:
+		row = License(license)
+		licenses[row.name] = row
+		session.add(row)
+
+	for license in ["CC-BY-NC-SA", "Other (Non-free)"]:
+		row = License(license, False)
+		licenses[row.name] = row
+		session.add(row)
+
+
+def populate_test_data(session):
+	licenses = { x.name : x for x in License.query.all() }
+	tags = { x.name : x for x in Tag.query.all() }
+	admin_user = User.query.filter_by(rank=UserRank.ADMIN).first()
+	v4 = MinetestRelease.query.filter_by(protocol=32).first()
+	v50 = MinetestRelease.query.filter_by(protocol=37).first()
+	v51 = MinetestRelease.query.filter_by(protocol=38).first()
 
-def defineDummyData(licenses, tags, ruben):
 	ez = User("Shara")
 	ez.github_username = "Ezhh"
 	ez.forums_username = "Shara"
 	ez.rank = UserRank.EDITOR
-	db.session.add(ez)
+	session.add(ez)
 
-	not1 = Notification(ruben, ez, "Awards approved", "/packages/rubenwardy/awards/")
-	db.session.add(not1)
+	not1 = Notification(admin_user, ez, "Awards approved", "/packages/rubenwardy/awards/")
+	session.add(not1)
 
 	jeija = User("Jeija")
 	jeija.github_username = "Jeija"
-	db.session.add(jeija)
+	jeija.forums_username = "Jeija"
+	session.add(jeija)
 
 
 	mod = Package()
@@ -44,36 +67,38 @@ def defineDummyData(licenses, tags, ruben):
 	mod.name = "alpha"
 	mod.title = "Alpha Test"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["mapgen"])
 	mod.tags.append(tags["environment"])
 	mod.repo = "https://github.com/ezhh/other_worlds"
 	mod.issueTracker = "https://github.com/ezhh/other_worlds/issues"
 	mod.forums = 16015
-	mod.shortDesc = "The content library should not be used yet as it is still in alpha"
+	mod.short_desc = "The content library should not be used yet as it is still in alpha"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/ezhh/handholds/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod1 = Package()
 	mod1.approved = True
 	mod1.name = "awards"
 	mod1.title = "Awards"
 	mod1.license = licenses["LGPLv2.1"]
+	mod1.media_license = licenses["MIT"]
 	mod1.type = PackageType.MOD
-	mod1.author = ruben
+	mod1.author = admin_user
 	mod1.tags.append(tags["player_effects"])
 	mod1.repo = "https://github.com/rubenwardy/awards"
 	mod1.issueTracker = "https://github.com/rubenwardy/awards/issues"
 	mod1.forums = 4870
-	mod1.shortDesc = "Adds achievements and an API to register new ones."
+	mod1.short_desc = "Adds achievements and an API to register new ones."
 	mod1.desc = """
 Majority of awards are back ported from Calinou's old fork in Carbone, under same license.
 
@@ -92,10 +117,11 @@ awards.register_achievement("award_mesefind",{
 
 	rel = PackageRelease()
 	rel.package = mod1
+	rel.min_rel = v51
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/rubenwardy/awards/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod2 = Package()
 	mod2.approved = True
@@ -104,22 +130,13 @@ awards.register_achievement("award_mesefind",{
 	mod2.tags.append(tags["tools"])
 	mod2.type = PackageType.MOD
 	mod2.license = licenses["LGPLv3"]
+	mod2.media_license = licenses["MIT"]
 	mod2.author = jeija
 	mod2.repo = "https://github.com/minetest-mods/mesecons/"
 	mod2.issueTracker = "https://github.com/minetest-mods/mesecons/issues"
 	mod2.forums = 628
-	mod2.shortDesc = "Mesecons adds everything digital, from all kinds of sensors, switches, solar panels, detectors, pistons, lamps, sound blocks to advanced digital circuitry like logic gates and programmable blocks."
+	mod2.short_desc = "Mesecons adds everything digital, from all kinds of sensors, switches, solar panels, detectors, pistons, lamps, sound blocks to advanced digital circuitry like logic gates and programmable blocks."
 	mod2.desc = """
-    ########################################################################
-    ##  __    __   _____   _____   _____   _____   _____   _   _   _____  ##
-    ## |  \  /  | |  ___| |  ___| |  ___| |  ___| |  _  | | \ | | |  ___| ##
-    ## |   \/   | | |___  | |___  | |___  | |     | | | | |  \| | | |___  ##
-    ## | |\__/| | |  ___| |___  | |  ___| | |     | | | | |     | |___  | ##
-    ## | |    | | | |___   ___| | | |___  | |___  | |_| | | |\  |  ___| | ##
-    ## |_|    |_| |_____| |_____| |_____| |_____| |_____| |_| \_| |_____| ##
-    ##                                                                    ##
-    ########################################################################
-
 MESECONS by Jeija and contributors
 
 Mezzee-what?
@@ -192,36 +209,39 @@ No warranty is provided, express or implied, for any part of the project.
 
 """
 
-	db.session.add(mod1)
-	db.session.add(mod2)
+	session.add(mod1)
+	session.add(mod2)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "handholds"
 	mod.title = "Handholds"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
 	mod.author = ez
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/ezhh/handholds"
 	mod.issueTracker = "https://github.com/ezhh/handholds/issues"
 	mod.forums = 17069
-	mod.shortDesc = "Adds hand holds and climbing thingies"
+	mod.short_desc = "Adds hand holds and climbing thingies"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
+	rel.max_rel = v4
 	rel.url = "https://github.com/ezhh/handholds/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "other_worlds"
 	mod.title = "Other Worlds"
 	mod.license = licenses["MIT"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
 	mod.author = ez
 	mod.tags.append(tags["mapgen"])
@@ -229,41 +249,42 @@ No warranty is provided, express or implied, for any part of the project.
 	mod.repo = "https://github.com/ezhh/other_worlds"
 	mod.issueTracker = "https://github.com/ezhh/other_worlds/issues"
 	mod.forums = 16015
-	mod.shortDesc = "Adds space with asteroids and comets"
+	mod.short_desc = "Adds space with asteroids and comets"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "food"
 	mod.title = "Food"
 	mod.license = licenses["LGPLv2.1"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/rubenwardy/food/"
 	mod.issueTracker = "https://github.com/rubenwardy/food/issues/"
 	mod.forums = 2960
-	mod.shortDesc = "Adds lots of food and an API to manage ingredients"
+	mod.short_desc = "Adds lots of food and an API to manage ingredients"
 	mod.desc = "This is the long desc"
-	food = mod
-	db.session.add(mod)
+	session.add(mod)
 
 	mod = Package()
 	mod.approved = True
 	mod.name = "food_sweet"
 	mod.title = "Sweet Foods"
 	mod.license = licenses["CC0"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.MOD
-	mod.harddeps.append(food)
-	mod.author = ruben
+	mod.author = admin_user
 	mod.tags.append(tags["player_effects"])
 	mod.repo = "https://github.com/rubenwardy/food_sweet/"
 	mod.issueTracker = "https://github.com/rubenwardy/food_sweet/issues/"
 	mod.forums = 9039
-	mod.shortDesc = "Adds sweet food"
+	mod.short_desc = "Adds sweet food"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	food_sweet = mod
+	session.add(mod)
 
 	game1 = Package()
 	game1.approved = True
@@ -271,28 +292,29 @@ No warranty is provided, express or implied, for any part of the project.
 	game1.title = "Capture The Flag"
 	game1.type = PackageType.GAME
 	game1.license = licenses["LGPLv2.1"]
-	game1.author = ruben
+	game1.media_license = licenses["MIT"]
+	game1.author = admin_user
 	game1.tags.append(tags["pvp"])
 	game1.tags.append(tags["survival"])
 	game1.tags.append(tags["multiplayer"])
 	game1.repo = "https://github.com/rubenwardy/capturetheflag"
 	game1.issueTracker = "https://github.com/rubenwardy/capturetheflag/issues"
 	game1.forums = 12835
-	game1.shortDesc = "Two teams battle to snatch and return the enemy's flag, before the enemy takes their own!"
+	game1.short_desc = "Two teams battle to snatch and return the enemy's flag, before the enemy takes their own!"
 	game1.desc = """
 As seen on the Capture the Flag server (minetest.rubenwardy.com:30000)
 
 Uses the CTF PvP Engine.
 """
 
-	db.session.add(game1)
+	session.add(game1)
 
 	rel = PackageRelease()
 	rel.package = game1
 	rel.title = "v1.0.0"
 	rel.url = "https://github.com/rubenwardy/capturetheflag/archive/master.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
 
 	mod = Package()
@@ -300,53 +322,33 @@ Uses the CTF PvP Engine.
 	mod.name = "pixelbox"
 	mod.title = "PixelBOX Reloaded"
 	mod.license = licenses["CC0"]
+	mod.media_license = licenses["MIT"]
 	mod.type = PackageType.TXP
-	mod.author = ruben
+	mod.author = admin_user
 	mod.forums = 14132
-	mod.shortDesc = "This is an update of the original PixelBOX texture pack by the brillant artist Gambit"
+	mod.short_desc = "This is an update of the original PixelBOX texture pack by the brillant artist Gambit"
 	mod.desc = "This is the long desc"
-	db.session.add(mod)
+	session.add(mod)
 
 	rel = PackageRelease()
 	rel.package = mod
 	rel.title = "v1.0.0"
 	rel.url = "http://mamadou3.free.fr/Minetest/PixelBOX.zip"
 	rel.approved = True
-	db.session.add(rel)
+	session.add(rel)
 
+	session.commit()
 
-delete_db = len(sys.argv) >= 2 and sys.argv[1].strip() == "-d"
-if delete_db and os.path.isfile("db.sqlite"):
-	os.remove("db.sqlite")
+	metas = {}
+	for package in Package.query.filter_by(type=PackageType.MOD).all():
+		meta = None
+		try:
+			meta = metas[package.name]
+		except KeyError:
+			meta = MetaPackage(package.name)
+			session.add(meta)
+			metas[package.name] = meta
+		package.provides.append(meta)
 
-print("Creating database tables...")
-db.create_all()
-print("Filling database...")
-
-ruben = User("rubenwardy")
-ruben.github_username = "rubenwardy"
-ruben.forums_username = "rubenwardy"
-ruben.rank = UserRank.ADMIN
-db.session.add(ruben)
-
-tags = {}
-for tag in ["Inventory", "Mapgen", "Building", \
-		"Mobs and NPCs", "Tools", "Player effects", \
-		"Environment", "Transport", "Maintenance", "Plants and farming", \
-		"PvP", "PvE", "Survival", "Creative", "Puzzle", "Multiplayer", "Singleplayer"]:
-	row = Tag(tag)
-	tags[row.name] = row
-	db.session.add(row)
-
-licenses = {}
-for license in ["GPLv2.1", "GPLv3", "LGPLv2.1", "LGPLv3", "AGPLv2.1", "AGPLv3",
-				"Apache", "BSD 3-Clause", "BSD 2-Clause", "CC0", "CC-BY-SA",
-				"CC-BY", "CC-BY-NC-SA", "MIT", "ZLib"]:
-	row = License(license)
-	licenses[row.name] = row
-	db.session.add(row)
-
-if test_data:
-	defineDummyData(licenses, tags, ruben)
-
-db.session.commit()
+	dep = Dependency(food_sweet, meta=metas["food"])
+	session.add(dep)

app/flatpages/help.md

@@ -1,4 +1,7 @@
 title: Help
 
-* [Ranks and Permissions](ranks_permissions)
 * [Package Tags](package_tags)
+* [Ranks and Permissions](ranks_permissions)
+* [Content Ratings and Flags](content_flags)
+* [Reporting Content](reporting)
+* [API](api)

app/flatpages/help/api.md

@@ -0,0 +1,51 @@
+title: API
+
+## Authentication
+
+Not all endpoints require authentication.
+Authentication is done using Bearer tokens:
+
+	Authorization: Bearer YOURTOKEN
+
+You can use the `/api/whoami` to check authentication.
+
+## Endpoints
+
+### Misc
+
+* GET `/api/whoami/` - Json dictionary with the following keys:
+	* `is_authenticated` - True on successful API authentication
+	* `username` - Username of the user authenticated as, null otherwise.
+	* 403 will be thrown on unsupported authentication type, invalid access token, or other errors.
+
+### Packages
+
+* GET `/api/packages/` - See [Package Queries](#package-queries)
+* GET `/api/packages/<username>/<name>/`
+
+### Topics
+
+* GET `/api/topics/` - Supports [Package Queries](#package-queries), and the following two options:
+    * `show_added` - Show topics which exist as packages, default true.
+	* `show_discarded` - Show topics which have been marked as outdated, default false.
+
+### Minetest
+
+* GET `/api/minetest_versions/`
+
+
+## Package Queries
+
+Example:
+
+	/api/packages/?type=mod&type=game&q=mobs+fun&hide=nonfree&hide=gore
+
+Supported query parameters:
+
+* `type` - Package types (`mod`, `game`, `txp`).
+* `q` - Query string
+* `random` - When present, enable random ordering and ignore `sort`.
+* `hide` - Hide content based on [Content Flags](content_flags).
+* `sort` - Sort by (`name`, `views`, `date`, `score`).
+* `order` - Sort ascending (`Asc`) or descending (`desc`).
+* `protocol_version` - Only show packages supported by this Minetest protocol version.

app/flatpages/help/content_flags.md

@@ -0,0 +1,26 @@
+title: Content Flags
+
+Content flags allow you to hide content based on your preferences.
+The filtering is done server-side, which means that you don't need to update
+your client to use new flags.
+
+## Flags
+
+* `nonfree` - can be used to hide packages which do not qualify as
+	'free software', as defined by the Free Software Foundation.
+* A content rating, given below.
+
+
+## Ratings
+
+Content ratings aren't currently supported by ContentDB.
+Instead, mature content isn't allowed at all for now.
+
+In the future, more mature content will be allowed but labelled with
+content ratings which may contain the following:
+
+* android_default - meta-rating which includes gore and drugs.
+* desktop_default - meta-rating which won't include anything for now.
+* gore - more than just blood
+* drugs
+* swearing

app/flatpages/help/package_tags.md

@@ -2,26 +2,32 @@ title: Package Tags
 
 ## Overview
 
-Tags should be added to packages to enable easy identification of different types of mods and games. 
+Tags should be added to packages to enable easy identification of different types of mods, games and texture packs. 
 
 They are only beneficial when applied correctly, so please use the following guidelines.
 
 ## Tag Usage
 
-* **Inventory** - For mods that add new inventory systems or new inventory pages.
-* **Mapgen** - For mods that add new biomes, new mapgen decorations, or any other mapgen elements.
 * **Building** - For mods that focus on adding new materials or nodes to build with.
-* **Mobs and NPCs** - For mods that add mobs or NPCs, or provide tools that assist with mob and NPC creation or manipulation.
-* **Tools** - For mods that add new tools or new features for existing tools.
-* **Player effects** - For mods that change player effects, for example speed, jump height or gravity.
+* **Education** - For mods or games created for educational purposes.
 * **Environment** - For mods that add environmental effects, including ambient sound and weather effects.
-* **Transport** - For mods that add transportation methods. This includes teleportation, vehicles and ridable mobs.
+* **Inventory** - For mods that add new inventory systems or new inventory pages.
+* **Machines and Electronics** - For mods that include placeable machinery or electronic components which interact to complete tasks.
 * **Maintenance** - For mods that assist with world or player maintenance. This includes large-scale map manipulation, area protection and other administrative tools.
-* **Plants and farming** - For mods that add new plants or other farmable resources. 
+* **Mapgen** - For mods that add new biomes, new mapgen decorations, or any other mapgen elements.
+* **Mobs and NPCs** - For mods that add mobs or NPCs, or provide tools that assist with mob and NPC creation or manipulation.
+* **Plants and Farming** - For mods that add new plants or other farmable resources. 
+* **Player effects/Food** - For mods that change player effects, for example speed, jump height or gravity, and food.
+* **Tools** - For mods that add new tools or new features for existing tools.
+* **Transport** - For mods that add transportation methods. This includes teleportation, vehicles and ridable mobs.
 * **Survival** - For mods written specifically for survival games. For example, these mods might focus on game-balance or increase the difficulty level. This tag should also be used for games with a heavy survival focus.
 * **Creative** - For mods written specifically (and often exclusively) for use in creative mode. For example, these mods may add a large amount of decorative content, or content without crafting recipes. This tag should also be used for games with a heavy creative focus.
+* **Multiplayer-focused** - For games that can be played with other players.
+* **Singleplayer-focused** - For games that can be played alone.
 * **PvP** - For games designed to be played competitively against other players.
 * **PvE** - For games designed for one or multiple players which focus on combat against mobs or NPCs.
-* **Puzzle** - For games with a focus on puzzle solving instead of combat.
-* **Multiplayer** - For games that can be played with other players.
-* **Singleplayer** - For games that can be played alone.
+* **Puzzle** - For mods and games with a focus on puzzle solving instead of combat.
+* **16px** - For 16px texture packs.
+* **32px** - For 32px texture packs.
+* **64px** - For 64px texture packs.
+* **128px+** - For 128px or higher texture packs.

app/flatpages/help/ranks_permissions.md

@@ -4,6 +4,7 @@ title: Ranks and Permissions
 
 * **New Members** - mostly untrusted, cannot change package meta data or publish releases without approval.
 * **Members** - Trusted to change the meta data of their own packages', but cannot publish releases.
+* **Trusted Members** - Same as above, but can approve their own releases and packages.
 * **Editors** - Trusted to change the meta data of any package, and also make and publish releases.
 * **Moderators** - Same as above, but can manage users.
 * **Admins** - Full access.
@@ -16,6 +17,7 @@ title: Ranks and Permissions
 			<th>Rank</th>
 			<th colspan=2>New Member</th>
 			<th colspan=2>Member</th>
+			<th colspan=2>Trusted Member</th>
 			<th colspan=2>Editor</th>
 			<th colspan=2>Moderator</th>
 			<th colspan=2>Admin</th>
@@ -32,6 +34,8 @@ title: Ranks and Permissions
 			<th>N</th>
 			<th>Y</th>
 			<th>N</th>
+			<th>Y</th>
+			<th>N</th>
 		</tr>
 	</thead>
 	<tbody>
@@ -41,6 +45,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -54,6 +60,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th></th> <!-- member -->
 			<th></th>
+			<th></th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -67,6 +75,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -80,6 +90,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -91,7 +103,9 @@ title: Ranks and Permissions
 			<td>Approve Screenshot</td>
 			<th></th> <!-- new -->
 			<th></th>
-			<th>✓</th> <!-- member -->
+			<th></th> <!-- member -->
+			<th></th>
+			<th>✓</th> <!-- trusted member -->
 			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
@@ -106,6 +120,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -119,6 +135,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -132,6 +150,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -145,6 +165,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th></th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th>✓</th>
 			<th>✓</th> <!-- moderator -->
@@ -158,6 +180,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th></th> <!-- member -->
 			<th></th>
+			<th></th> <!-- trusted member -->
+			<th></th>
 			<th></th> <!-- editor -->
 			<th></th>
 			<th></th> <!-- moderator -->
@@ -165,12 +189,44 @@ title: Ranks and Permissions
 			<th>✓</th> <!-- admin -->
 			<th>✓</th>
 		</tr>
+		<tr>
+			<td>See Private Thread</td>
+			<th>✓</th> <!-- new -->
+			<th></th>
+			<th>✓</th> <!-- member -->
+			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
+			<th>✓</th> <!-- editor -->
+			<th>✓</th>
+			<th>✓</th> <!-- moderator -->
+			<th>✓</th>
+			<th>✓</th> <!-- admin -->
+			<th>✓</th>
+		</tr>
 		<tr>
 			<td>Set Email</td>
 			<th>✓</th> <!-- new -->
 			<th></th>
 			<th>✓</th> <!-- member -->
 			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
+			<th>✓</th> <!-- editor -->
+			<th></th>
+			<th>✓</th> <!-- moderator -->
+			<th>✓<sup>2</sup></th>
+			<th>✓</th> <!-- admin -->
+			<th>✓</th>
+		</tr>
+		<tr>
+			<td>Create Token</td>
+			<th></th> <!-- new -->
+			<th></th>
+			<th>✓</th> <!-- member -->
+			<th></th>
+			<th>✓</th> <!-- trusted member -->
+			<th></th>
 			<th>✓</th> <!-- editor -->
 			<th></th>
 			<th>✓</th> <!-- moderator -->
@@ -184,6 +240,8 @@ title: Ranks and Permissions
 			<th></th>
 			<th></th> <!-- member -->
 			<th></th>
+			<th></th> <!-- trusted member -->
+			<th></th>
 			<th></th> <!-- editor -->
 			<th></th>
 			<th>✓<sup>3</sup></th> <!-- moderator -->

app/flatpages/help/reporting.md

@@ -0,0 +1,8 @@
+title: Reporting Content
+
+Please let us know if anything on the ContentDB violates our rules or any applicable
+laws.
+
+We take copyright violation and other offenses very seriously.
+
+<a href="https://rubenwardy.com/contact/" class="btn btn-success">Contact</a>

app/flatpages/help/wtfpl.md

@@ -0,0 +1,42 @@
+title: WTFPL is a terrible license
+no_h1: true
+
+<div id="warning" class="alert alert-warning">
+	<span class="icon_message"></span>
+
+	Please reconsider the choice of WTFPL as a license.
+
+	<script src="/static/jquery.min.js"></script>
+	<script>
+		// @author rubenwardy
+		// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+
+		var params = new URLSearchParams(location.search);
+		var r      = params.get("r");
+		if (r)
+			document.write("<a class='alert_right button' href='" + r + "'>Okay</a>");
+		else
+			$("#warning").hide();
+	</script>
+</div>
+
+# WTFPL is a terrible license
+
+The use of WTFPL as a license is discouraged for multiple reasons.
+
+* **No Warranty disclaimer:** This could open you up to being sued.<sup>[1]</sup>
+* **Swearing:** This prevents settings like schools from using your content.
+* **Not OSI Approved:** Same as public domain?
+
+The Open Source Initiative chose not to approve the license as an open-source
+license, saying:<sup>[3]</sup>
+
+> It's no different from dedication to the public domain.
+> Author has submitted license approval request – author is free to make public domain dedication.
+> Although he agrees with the recommendation, Mr. Michlmayr notes that public domain doesn't exist in Europe. Recommend: Reject.
+
+## Sources
+
+1. [WTFPL is harmful to software developers](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html)
+2. [FSF](https://www.gnu.org/licenses/license-list.en.html)
+3. [OSI](https://opensource.org/minutes20090304)

app/flatpages/policy_and_guidance.md

@@ -0,0 +1,140 @@
+title: Package Inclusion Policy and Guidance
+
+<div class="alert alert-warning">
+	<b>Note:</b> This is a draft
+</div>
+
+## 0. Overview
+
+ContentDB is for the community, and as such listings should be useful to the
+community. To help with this, there are a few rules to improve the quality of
+the listings and to combat abuse.
+
+* No inappropriate content. <sup>2.1</sup>
+* Content must be playable/useful, but not necessarily finished. <sup>2.2</sup>
+* Don't use the name of another mod unless your mod is a fork or reimplementation. <sup>3</sup>
+* Licenses must allow derivatives, redistribution, and must not discriminate. <sup>4</sup>
+* Don't put promotions or advertisements in package listings, except for
+  donation and personal website links which are permitted in the
+  long description. <sup>5</sup>
+* The ContentDB admin reserves the right to remove packages for any reason,
+  including ones not covered by this document, and to ban users who abuse
+  this service. <sup>1</sup>
+
+
+## 1. General
+
+The ContentDB admin reserves the right to remove packages for any reason,
+including ones not covered by this document, and to ban users who abuse this service.
+
+Also see the [help page on tags](/help/package_tags/).
+
+
+## 2. Accepted Content
+
+### 2.1. Acceptable Content
+
+Sexually-orientated content is not permitted.
+
+Mature content, including that relating to drugs, excessive gore, violence, or
+excessive horror, is not currently permitted - but will be in the future.
+
+The submission of malware is strictly prohibited. This includes software which
+does not do as it advertises, for example if it posts telemetry without stating
+clearly that it does in the package meta.
+
+### 2.2. State of Completion
+
+ContentDB should only currently contain playable content - content which is
+sufficiently complete to be useful to end users. It's fine to add stuff which
+is still a work in progress (WIP) as long as it adds sufficient value -
+MineClone 2 is a good example of a WIP package which may break between releases
+but still has value. Note that this doesn't mean that you should add a thing
+you started working on yesterday, it's worth adding all the basic stuff to
+make your package useful.
+
+Adding non-player facing mods, such as libraries and server tools, is perfectly fine
+and encouraged. ContentDB isn't just for player-facing things, and adding
+libraries allows them to be installed when a mod depends on it.
+
+
+## 3. Technical Names
+
+### 3.1 Right to a name
+
+The first package to use a name based on the creation of its forum topic or
+contentdb submission has the right to the technical name. The use of a package
+on a server or in private doesn't reserve its name. No other packages of the same
+type may use the same name, except for the exception given by 3.2.
+
+If it turns out that we made a mistake by approving a package and that the
+name should have been given to another package, then we *may* unapprove the
+package and give the name to the correct one.
+
+If you submit a package where you don't have the right to the name you will be asked
+to change the name of the package, or your package won't be accepted.
+
+We reserve the right to issue exceptions for this where we feel necessary.
+
+### 3.2 Mod Forks and Reimplementations
+
+An exception to the above is that mods are allowed to have the same name as a
+mod if its a fork of that mod (or a close reimplementation). In real terms, it
+should be possible to use the new mod as a drop-in replacement.
+
+We reserve the right to decide whether a mod counts as a fork or
+reimplementation of the mod that owns the name.
+
+
+## 4. Licenses
+
+### 4.1 Allowed Licenses
+
+Please ensure that you correctly credit any resources (code, assets, or otherwise)
+that you have used in your package.
+
+**The use of licenses which do not allow derivatives or redistribution is not
+permitted. This includes CC-ND (No-Derivatives) and lots of closed source licenses.
+The use of licenses which discriminate between groups of people or forbid the use
+of the content on servers or singleplayer is also not permitted.**
+
+However, closed sourced licenses are allowed if they allow the above.
+
+If the license you use is not on the list then please select "Other", and we'll
+get around to adding it.
+
+Please note that the definitions of "free" and "non-free" is the same as that
+of the [Free Software Foundation](https://www.gnu.org/philosophy/free-sw.en.html).
+
+### 4.2 Recommended Licenses
+
+It is highly recommended that you use a free and open source software license.
+FOSS licenses result in a sharing community and will increase the number of potential users your package has.
+Using a closed source license will result in your package being massively penalised in the search results and package lists.
+
+It is recommended that you use a proper license for code with a warranty
+disclaimer, such as the (L)GPL or MIT. You should also use a proper media license
+for media, such as a Creative Commons license.
+
+The use of WTFPL is discouraged as it doesn't contain a [valid warranty disclaimer](https://cubicspot.blogspot.com/2017/04/wtfpl-is-harmful-to-software-developers.html),
+and also includes swearing which prevents settings like schools from using your content.
+[Read more](/help/wtfpl/).
+
+Public domain is not a valid license in many countries, please use CC0 or MIT instead.
+
+
+## 5. Promotions and Advertisements (inc. asking for donations)
+
+You may note place any promotions or advertisements in any meta data including
+screensthos. This includes asking for donations, promoting online shops,
+or linking to personal websites and social media. Please instead use the
+fields provided on your user profile page to place links to websites and
+donation pages.
+
+ContentDB is for the community. We may remove any promotions if we feel that
+they're inappropriate.
+
+
+## 6. Reporting Violations
+
+See the [Reporting Content](/help/reporting/) page.

app/maillogger.py

@@ -0,0 +1,109 @@
+import logging
+from enum import Enum
+from app.tasks.emails import sendEmailRaw
+
+def _has_newline(line):
+	"""Used by has_bad_header to check for \\r or \\n"""
+	if line and ("\r" in line or "\n" in line):
+		return True
+	return False
+
+def _is_bad_subject(subject):
+	"""Copied from: flask_mail.py class Message def has_bad_headers"""
+	if _has_newline(subject):
+		for linenum, line in enumerate(subject.split("\r\n")):
+			if not line:
+				return True
+			if linenum > 0 and line[0] not in "\t ":
+				return True
+			if _has_newline(line):
+				return True
+			if len(line.strip()) == 0:
+				return True
+	return False
+
+
+class FlaskMailSubjectFormatter(logging.Formatter):
+	def format(self, record):
+		record.message = record.getMessage()
+		if self.usesTime():
+			record.asctime = self.formatTime(record, self.datefmt)
+		s = self.formatMessage(record)
+		return s
+
+class FlaskMailTextFormatter(logging.Formatter):
+	pass
+
+# TODO: hier nog niet tevreden over (vooral logger.error(..., exc_info, stack_info))
+class FlaskMailHTMLFormatter(logging.Formatter):
+	pre_template = "<h1>%s</h1><pre>%s</pre>"
+	def formatException(self, exc_info):
+		formatted_exception = logging.Handler.formatException(self, exc_info)
+		return FlaskMailHTMLFormatter.pre_template % ("Exception information", formatted_exception)
+	def formatStack(self, stack_info):
+		return FlaskMailHTMLFormatter.pre_template % ("<h1>Stack information</h1><pre>%s</pre>", stack_info)
+
+
+# see: https://github.com/python/cpython/blob/3.6/Lib/logging/__init__.py (class Handler)
+
+class FlaskMailHandler(logging.Handler):
+	def __init__(self, mailer, subject_template, level=logging.NOTSET):
+		logging.Handler.__init__(self, level)
+		self.mailer = mailer
+		self.send_to = mailer.app.config["MAIL_UTILS_ERROR_SEND_TO"]
+		self.subject_template = subject_template
+		self.html_formatter = None
+
+	def setFormatter(self, text_fmt, html_fmt=None):
+		"""
+		Set the formatters for this handler. Provide at least one formatter.
+		When no text_fmt is provided, no text-part is created for the email body.
+		"""
+		assert (text_fmt, html_fmt) != (None, None), "At least one formatter should be provided"
+		if type(text_fmt)==str:
+			text_fmt = FlaskMailTextFormatter(text_fmt)
+		self.formatter = text_fmt
+		if type(html_fmt)==str:
+			html_fmt = FlaskMailHTMLFormatter(html_fmt)
+		self.html_formatter = html_fmt
+
+	def getSubject(self, record):
+		fmt = FlaskMailSubjectFormatter(self.subject_template)
+		subject = fmt.format(record)
+		#Since templates can cause header problems, and we rather have a incomplete email then an error, we fix this
+		if _is_bad_subject(subject):
+			subject="FlaskMailHandler log-entry from %s [original subject is replaced, because it would result in a bad header]" % self.mailer.app.name
+		return subject
+
+	def emit(self, record):
+		text = self.format(record)				if self.formatter	  else None
+		html = self.html_formatter.format(record) if self.html_formatter else None
+		sendEmailRaw.delay(self.send_to, self.getSubject(record), text, html)
+
+
+def register_mail_error_handler(app, mailer):
+	subject_template = "ContentDB crashed (%(module)s > %(funcName)s)"
+	text_template = """
+Message type: %(levelname)s
+Location:	 %(pathname)s:%(lineno)d
+Module:	   %(module)s
+Function:	 %(funcName)s
+Time:		 %(asctime)s
+Message:
+%(message)s"""
+	html_template = """
+<style>th { text-align: right}</style><table>
+<tr><th>Message type:</th><td>%(levelname)s</td></tr>
+<tr>	<th>Location:</th><td>%(pathname)s:%(lineno)d</td></tr>
+<tr>	  <th>Module:</th><td>%(module)s</td></tr>
+<tr>	<th>Function:</th><td>%(funcName)s</td></tr>
+<tr>		<th>Time:</th><td>%(asctime)s</td></tr>
+</table>
+<h2>Message</h2>
+<pre>%(message)s</pre>"""
+
+	import logging
+	mail_handler = FlaskMailHandler(mailer, subject_template)
+	mail_handler.setLevel(logging.ERROR)
+	mail_handler.setFormatter(text_template, html_template)
+	app.logger.addHandler(mail_handler)

app/public/static/opensearch.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
+	<ShortName>ContentDB</ShortName>
+	<LongName>ContentDB</LongName>
+	<InputEncoding>UTF-8</InputEncoding>
+	<Description>Search mods, games, and textures for Minetest.</Description>
+	<Tags>Minetest Mod Game Subgame Search</Tags>
+	<Url type="text/html" method="get" template="https://content.minetest.net/packages?q={searchTerms}"/>
+</OpenSearchDescription>

app/public/static/package_create.js

@@ -9,41 +9,49 @@ $(function() {
 		$(".pkg_meta").show()
 	}
 
-	function repoIsSupported(url) {
-		try {
-			return URI(url).hostname() == "github.com"
-		} catch(e) {
-			return false
-		}
-	}
-
 	$(".pkg_meta").hide()
 	$(".pkg_wiz_1").show()
+
+	$("#pkg_wiz_1_skip").click(finish)
 	$("#pkg_wiz_1_next").click(function() {
 		const repoURL = $("#repo").val();
-		if (repoIsSupported(repoURL)) {
+		if (repoURL.trim() != "") {
 			$(".pkg_wiz_1").hide()
 			$(".pkg_wiz_2").show()
 			$(".pkg_repo").hide()
 
-			performTask("/tasks/getmeta/new/?url=" + encodeURI(repoURL)).then(function(result) {
-				$("#name").val(result.name || "")
-				$("#title").val(result.title || "")
-				$("#repo").val(result.repo || repoURL)
-				$("#issueTracker").val(result.issueTracker || "")
-				$("#desc").val(result.description || "")
-				$("#shortDesc").val(result.short_description || "")
-				if (result.forumId) {
-					$("#forums").val(result.forumId)
+			function setField(id, value) {
+				if (value && value != "") {
+					var ele = $(id);
+					ele.val(value);
+					ele.trigger("change");
 				}
-				finish()
+			}
+
+			performTask("/tasks/getmeta/new/?url=" + encodeURI(repoURL)).then(function(result) {
+				setField("#name", result.name);
+				setField("#provides_str", result.provides);
+				setField("#title", result.title);
+				setField("#repo", result.repo || repoURL);
+				setField("#issueTracker", result.issueTracker);
+				setField("#desc", result.description);
+				setField("#short_desc", result.short_description);
+				setField("#harddep_str", result.depends);
+				setField("#softdep_str", result.optional_depends);
+				setField("#short_desc", result.short_description);
+				setField("#forums", result.forumId);
+				if (result.type && result.type.length > 2) {
+					$("#type").val(result.type);
+				}
+
+				finish();
 			}).catch(function(e) {
-				alert(e)
-				$(".pkg_wiz_1").show()
-				$(".pkg_wiz_2").hide()
-				$(".pkg_repo").show()
+				alert(e);
+				$(".pkg_wiz_1").show();
+				$(".pkg_wiz_2").hide();
+				$(".pkg_repo").show();
 				// finish()
-			})
+			});
 		} else {
 			finish()
 		}

app/public/static/package_edit.js

@@ -0,0 +1,64 @@
+// @author rubenwardy
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+
+$(function() {
+	$("#type").change(function() {
+		$(".not_mod, .not_game, .not_txp").show()
+		$(".not_" + this.value.toLowerCase()).hide()
+	})
+	$(".not_mod, .not_game, .not_txp").show()
+	$(".not_" + $("#type").val().toLowerCase()).hide()
+
+	$("#forums").on('paste', function(e) {
+		try {
+			var pasteData = e.originalEvent.clipboardData.getData('text')
+			var url = new URL(pasteData);
+			if (url.hostname == "forum.minetest.net") {
+				$(this).val(url.searchParams.get("t"));
+				e.preventDefault();
+			}
+		} catch (e) {
+			console.log("Not a URL");
+		}
+	});
+
+	let hint = null;
+	function showHint(ele, text) {
+		if (hint) {
+			hint.remove();
+		}
+
+		hint = ele.parent()
+				.append(`<div class="alert alert-warning my-3">${text}</div>`)
+				.find(".alert");
+	}
+
+	let hint_mtmods = `Tip:
+		Don't include <i>Minetest</i>, <i>mod</i>, or <i>modpack</i> anywhere in the short description.
+		It is unnecessary and wastes characters.`;
+
+	let hint_thegame = `Tip:
+		It's obvious that this adds something to Minetest,
+		there's no need to use phrases such as \"adds X to the game\".`
+
+	$("#short_desc").on("change paste keyup", function() {
+		var val = $(this).val().toLowerCase();
+		if (val.indexOf("minetest") >= 0 || val.indexOf("mod") >= 0 ||
+				val.indexOf("modpack") >= 0 || val.indexOf("mod pack") >= 0) {
+			showHint($(this), hint_mtmods);
+		} else if (val.indexOf("the game") >= 0) {
+			showHint($(this), hint_thegame);
+		} else if (hint) {
+			hint.remove();
+			hint = null;
+		}
+	})
+
+	var btn = $("#forums").parent().find("label").append("<a class='ml-3 btn btn-sm btn-primary'>Open</a>");
+	btn.click(function() {
+		var id = $("#forums").val();
+		if (/^\d+$/.test(id)) {
+			window.open("https://forum.minetest.net/viewtopic.php?t=" + id, "_blank");
+		}
+	});
+})

app/public/static/polltask.js

@@ -22,7 +22,7 @@ function pollTask(poll_url, disableTimeout) {
 		var tries = 0;
 		function retry() {
 			tries++;
-			if (!disableTimeout && tries > 10) {
+			if (!disableTimeout && tries > 30) {
 				reject("timeout")
 			} else {
 				const interval = Math.min(tries*100, 1000)

app/public/static/release_minmax.js

@@ -0,0 +1,18 @@
+var min = $("#min_rel");
+var max = $("#max_rel");
+var none = $("#min_rel option:first-child").attr("value");
+var warning = $("#minmax_warning");
+
+function ver_check() {
+	var minv = min.val();
+	var maxv = max.val();
+
+	if (minv != none && maxv != none && minv > maxv) {
+		warning.show();
+	} else {
+		warning.hide();
+	}
+}
+
+min.change(ver_check);
+max.change(ver_check);

app/public/static/tagselector.js

@@ -5,13 +5,25 @@
  * https://petprojects.googlecode.com/svn/trunk/GPL-LICENSE.txt
  */
 (function($) {
-	$.fn.tagSelector = function(source, name, select) {
+	function hide_error(input) {
+		var err = input.parent().parent().find(".invalid-remaining");
+		err.hide();
+	}
+
+	function show_error(input, msg) {
+		var err = input.parent().parent().find(".invalid-remaining");
+		console.log(err.length);
+		err.text(msg);
+		err.show();
+	}
+
+	$.fn.selectSelector = function(source, name, select) {
 		return this.each(function() {
 				var selector = $(this),
 					input = $('input[type=text]', this);
 
 				selector.click(function() { input.focus(); })
-					.delegate('.tag a', 'click', function() {
+					.delegate('.badge a', 'click', function() {
 						var id = $(this).parent().data("id");
 						for (var i = 0; i < source.length; i++) {
 							if (source[i].id == id) {
@@ -23,13 +35,14 @@
 					});
 
 				function addTag(item) {
-					var tag = $('<span class="tag"/>')
+					var tag = $('<span class="badge badge-pill badge-primary"/>')
 						.text(item.toString() + ' ')
 						.data("id", item.id)
 						.append('<a>x</a>')
 						.insertBefore(input);
 					input.attr("placeholder", null);
 					select.find("option[value=" + item.id + "]").attr("selected", "selected")
+					hide_error(input);
 				}
 
 				function recreate() {
@@ -42,6 +55,13 @@
 				}
 				recreate();
 
+				input.focusout(function(e) {
+					var value = input.val().trim()
+					if (value != "") {
+						show_error(input, "Please select an existing tag, it;s not possible to add custom ones.");
+					}
+				})
+
 				input.keydown(function(e) {
 						if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
 							e.preventDefault();
@@ -80,15 +100,149 @@
 			});
 	}
 
+	$.fn.csvSelector = function(source, name, result, allowSlash) {
+		return this.each(function() {
+				var selector = $(this),
+					input = $('input[type=text]', this);
+
+				var selected = [];
+				var lookup = {};
+				for (var i = 0; i < source.length; i++) {
+					lookup[source[i].id] = source[i];
+				}
+
+				selector.click(function() { input.focus(); })
+					.delegate('.badge a', 'click', function() {
+						var id = $(this).parent().data("id");
+						for (var i = 0; i < selected.length; i++) {
+							if (selected[i] == id) {
+								selected.splice(i, 1);
+							}
+						}
+						recreate();
+					});
+
+				function selectItem(id) {
+					for (var i = 0; i < selected.length; i++) {
+						if (selected[i] == id) {
+							return false;
+						}
+					}
+					selected.push(id);
+					return true;
+				}
+
+				function addTag(id, value) {
+					var tag = $('<span class="badge badge-pill badge-primary"/>')
+						.text(value)
+						.data("id", id)
+						.append(' <a>x</a>')
+						.insertBefore(input);
+
+					input.attr("placeholder", null);
+					hide_error(input);
+				}
+
+				function recreate() {
+					selector.find("span").remove();
+					for (var i = 0; i < selected.length; i++) {
+						var value = lookup[selected[i]] || { value: selected[i] };
+						addTag(selected[i], value.value);
+					}
+					result.val(selected.join(","))
+				}
+
+				function readFromResult() {
+					selected = [];
+					var selected_raw = result.val().split(",");
+					for (var i = 0; i < selected_raw.length; i++) {
+						var raw = selected_raw[i].trim();
+						if (lookup[raw] || raw.match(/^([a-z0-9_]+)$/)) {
+							selected.push(raw);
+						}
+					}
+
+					recreate();
+				}
+				readFromResult();
+
+				result.change(readFromResult);
+
+				input.focusout(function() {
+					var item = input.val();
+					if (item.length == 0) {
+						input.data("ui-autocomplete").search("");
+					} else if (item.match(/^([a-z0-9_]+)$/)) {
+						selectItem(item);
+						recreate();
+						input.val("");
+					}
+					return true;
+				});
+
+				input.keydown(function(e) {
+						if (e.keyCode === $.ui.keyCode.TAB && $(this).data('ui-autocomplete').menu.active)
+							e.preventDefault();
+						else if (e.keyCode === $.ui.keyCode.COMMA) {
+							var item = input.val();
+							if (item.length == 0) {
+								input.data("ui-autocomplete").search("");
+							} else if (item.match(/^([a-z0-9_]+)$/)) {
+								selectItem(item);
+								recreate();
+								input.val("");
+							} else {
+								show_error(input, "Only lowercase alphanumeric and number names allowed.");
+							}
+							e.preventDefault();
+							return true;
+						} else if (e.keyCode === $.ui.keyCode.BACKSPACE) {
+							if (input.val() == "") {
+								var item = selected[selected.length - 1];
+								selected.splice(selected.length - 1, 1);
+								recreate();
+								if (!(item.indexOf("/") > 0))
+									input.val(item);
+								e.preventDefault();
+								return true;
+							}
+						}
+					})
+					.autocomplete({
+						minLength: 0,
+						source: source,
+						select: function(event, ui) {
+							selectItem(ui.item.id);
+							recreate();
+							input.val("");
+							return false;
+						}
+					});
+
+				input.data('ui-autocomplete')._renderItem = function(ul, item) {
+						return $('<li/>')
+							.data('item.autocomplete', item)
+							.append($('<a/>').text(item.toString()))
+							.appendTo(ul);
+					};
+
+				input.data('ui-autocomplete')._resizeMenu = function(ul, item) {
+						var ul = this.menu.element;
+						ul.outerWidth(Math.max(
+							ul.width('').outerWidth(),
+							selector.outerWidth()
+						));
+					};
+			});
+	}
+
 	$(function() {
 		$(".multichoice_selector").each(function() {
 			var ele = $(this);
 			var sel = ele.parent().find("select");
-			console.log(sel.attr("name"));
-			sel.css("display", "none");
+			sel.hide();
 
 			var options = [];
-
 			sel.find("option").each(function() {
 				var text = $(this).text();
 				options.push({
@@ -100,7 +254,19 @@
 			});
 
 			console.log(options);
-			ele.tagSelector(options, sel.attr("name"), sel);
-		})
+			ele.selectSelector(options, sel.attr("name"), sel);
+		});
+
+		$(".metapackage_selector").each(function() {
+			var input = $(this).parent().children("input[type='text']");
+			input.hide();
+			$(this).csvSelector(meta_packages, input.attr("name"), input);
+		});
+
+		$(".deps_selector").each(function() {
+			var input = $(this).parent().children("input[type='text']");
+			input.hide();
+			$(this).csvSelector(all_packages, input.attr("name"), input);
+		});
 	});
 })(jQuery);

app/public/static/topic_discard.js

@@ -0,0 +1,29 @@
+$(".topic-discard").click(function() {
+	var ele = $(this);
+	var tid = ele.attr("data-tid");
+	var discard = !ele.parent().parent().hasClass("discardtopic");
+	fetch(new Request("/api/topic_discard/?tid=" + tid +
+			"&discard=" + (discard ? "true" : "false"), {
+		method: "post",
+		credentials: "same-origin",
+		headers: {
+			"Accept": "application/json",
+			"X-CSRFToken": csrf_token,
+		},
+	})).then(function(response) {
+		response.text().then(function(txt) {
+			console.log(JSON.parse(txt));
+			if (JSON.parse(txt).discarded) {
+				ele.parent().parent().addClass("discardtopic");
+				ele.removeClass("btn-danger");
+				ele.addClass("btn-success");
+				ele.text("Show");
+			} else {
+				ele.parent().parent().removeClass("discardtopic");
+				ele.removeClass("btn-success");
+				ele.addClass("btn-danger");
+				ele.text("Discard");
+			}
+		}).catch(console.log)
+	}).catch(console.log)
+});

app/querybuilder.py

@@ -0,0 +1,130 @@
+from .models import db, PackageType, Package, ForumTopic, License, MinetestRelease, PackageRelease
+from .utils import isNo, isYes
+from sqlalchemy.sql.expression import func
+from flask import abort
+from sqlalchemy import or_
+
+class QueryBuilder:
+	title  = None
+	types  = None
+	search = None
+
+	def __init__(self, args):
+		title = "Packages"
+
+		# Get request types
+		types = args.getlist("type")
+		types = [PackageType.get(tname) for tname in types]
+		types = [type for type in types if type is not None]
+		if len(types) > 0:
+			title = ", ".join([type.value + "s" for type in types])
+
+		hide_flags = args.getlist("hide")
+
+		self.title  = title
+		self.types  = types
+		self.search = args.get("q")
+		self.random = "random" in args
+		self.lucky  = "lucky" in args
+		self.hide_nonfree = "nonfree" in hide_flags
+		self.limit  = 1 if self.lucky else None
+		self.order_by  = args.get("sort")
+		self.order_dir = args.get("order") or "desc"
+		self.protocol_version = args.get("protocol_version")
+
+		self.show_discarded = isYes(args.get("show_discarded"))
+		self.show_added = args.get("show_added")
+		if self.show_added is not None:
+			self.show_added = isYes(self.show_added)
+
+		if self.search is not None and self.search.strip() == "":
+			self.search = None
+
+	def setSortIfNone(self, name):
+		if self.order_by is None:
+			self.order_by = name
+
+	def getMinetestVersion(self):
+		if not self.protocol_version:
+			return None
+
+		self.protocol_version = int(self.protocol_version)
+		version = MinetestRelease.query.filter(MinetestRelease.protocol>=self.protocol_version).first()
+		if version is not None:
+			return version.id
+		else:
+			return 10000000
+
+	def buildPackageQuery(self):
+		query = Package.query.filter_by(soft_deleted=False, approved=True)
+
+		if len(self.types) > 0:
+			query = query.filter(Package.type.in_(self.types))
+
+		if self.search:
+			query = query.search(self.search, sort=True)
+
+		if self.random:
+			query = query.order_by(func.random())
+		else:
+			to_order = None
+			if self.order_by is None or self.order_by == "score":
+				to_order = Package.score
+			elif self.order_by == "created_at":
+				to_order = Package.created_at
+			else:
+				abort(400)
+
+			if self.order_dir == "asc":
+				to_order = db.asc(to_order)
+			elif self.order_dir == "desc":
+				to_order = db.desc(to_order)
+			else:
+				abort(400)
+
+			query = query.order_by(to_order)
+
+		if self.hide_nonfree:
+			query = query.filter(Package.license.has(License.is_foss == True))
+			query = query.filter(Package.media_license.has(License.is_foss == True))
+
+		if self.protocol_version:
+			version = self.getMinetestVersion()
+			query = query.join(Package.releases) \
+				.filter(PackageRelease.approved==True) \
+				.filter(or_(PackageRelease.min_rel_id==None, PackageRelease.min_rel_id<=version)) \
+				.filter(or_(PackageRelease.max_rel_id==None, PackageRelease.max_rel_id>=version))
+
+		if self.limit:
+			query = query.limit(self.limit)
+
+		return query
+
+	def buildTopicQuery(self, show_added=False):
+		query = ForumTopic.query
+
+		if not self.show_discarded:
+			query = query.filter_by(discarded=False)
+
+		show_added = self.show_added == True or (self.show_added is None and show_added)
+		if not show_added:
+			query = query.filter(~ db.exists().where(Package.forums==ForumTopic.topic_id))
+
+		if self.order_by is None or self.order_by == "name":
+			query = query.order_by(db.asc(ForumTopic.wip), db.asc(ForumTopic.name), db.asc(ForumTopic.title))
+		elif self.order_by == "views":
+			query = query.order_by(db.desc(ForumTopic.views))
+		elif self.order_by == "date":
+			query = query.order_by(db.asc(ForumTopic.created_at))
+			sort_by = "date"
+
+		if self.search:
+			query = query.filter(ForumTopic.title.ilike('%' + self.search + '%'))
+
+		if len(self.types) > 0:
+			query = query.filter(ForumTopic.type.in_(self.types))
+
+		if self.limit:
+			query = query.limit(self.limit)
+
+		return query

app/rediscache.py

@@ -0,0 +1,13 @@
+from . import r
+
+# This file acts as a facade between the releases code and redis,
+# and also means that the releases code avoids knowing about `app`
+
+def make_download_key(ip, package):
+	return ("{}/{}/{}").format(ip, package.author.username, package.name)
+
+def set_key(key, v):
+	r.set(key, v)
+
+def has_key(key):
+	return r.exists(key)

app/sass.py

@@ -15,8 +15,6 @@ import codecs
 from flask import *
 from scss import Scss
 
-from app import app
-
 def _convert(dir, src, dst):
 	original_wd = os.getcwd()
 	os.chdir(dir)
@@ -31,7 +29,7 @@ def _convert(dir, src, dst):
 	outfile.write(output)
 	outfile.close()
 
-def _getDirPath(originalPath, create=False):
+def _getDirPath(app, originalPath, create=False):
 	path = originalPath
 
 	if not os.path.isdir(path):
@@ -47,8 +45,8 @@ def _getDirPath(originalPath, create=False):
 
 def sass(app, inputDir='scss', outputPath='static', force=False, cacheDir="public/static"):
 	static_url_path = app.static_url_path
-	inputDir = _getDirPath(inputDir)
-	cacheDir = _getDirPath(cacheDir or outputPath, True)
+	inputDir = _getDirPath(app, inputDir)
+	cacheDir = _getDirPath(app, cacheDir or outputPath, True)
 
 	def _sass(filepath):
 		sassfile = "%s/%s.scss" % (inputDir, filepath)
@@ -63,5 +61,3 @@ def sass(app, inputDir='scss', outputPath='static', force=False, cacheDir="publi
 		return send_from_directory(cacheDir, filepath + ".css")
 
 	app.add_url_rule("/%s/<path:filepath>.css" % (outputPath), 'sass', _sass)
-
-sass(app)

app/scss/comments.scss

@@ -0,0 +1,29 @@
+.img-thumbnail-1 {
+	padding: 0px;
+	// width: 100%;
+	background: white;
+}
+
+.comments {
+	list-style: none;
+	padding: 0;
+
+	.card {
+		position:relative;
+
+		.card-header:before {
+			position: absolute;
+			top: 11px;
+			right: 100%;
+			width: 0;
+			height: 0;
+			display: block;
+			content:" ";
+			border-color: transparent;
+			border-style: solid solid outset;
+			pointer-events:none;
+			border-right-color: #444;
+			border-width: 14px;
+		}
+	}
+}

app/scss/components.scss

@@ -1,108 +1,3 @@
-h1 {
-	margin: 0;
-}
-
-h2, h3 {
-	margin: 5px 0;
-}
-
-a {
-	color: #0be;
-	font-weight: bold;
-	text-decoration: none;
-}
-
-a:hover {
-	color: #0df;
-	text-decoration: underline;
-}
-
-/* Containers */
-
-.box {
-	border-radius: 5px;
-	margin: 15px 0;
-	padding: 0;
-}
-
-.box h2, .box h3 {
-	margin: 0;
-	padding: 0.5em 0.5em 0.5em 15px;
-	border-bottom: 1px solid #444;
-}
-
-.box .box-body {
-	padding: 1em !important;
-}
-
-// .box form {
-// 	padding: 1em;
-// }
-
-.box_grey {
-	background: #333;
-	border: 1px solid #444;
-}
-
-.ul_boxes {
-	display: block;
-	margin: 0;
-	padding: 0;
-	list-style: none;
-}
-
-.ul_boxes > li {
-	padding: 0;
-	list-style: none;
-}
-
-.box_link {
-	display: block;
-	color: #ddd;
-	text-decoration: none;
-}
-
-.box_link:hover{
-	background: #3a3a3a;
-}
-
-
-/*
-	buttonset
-*/
-
-.buttonset, .buttonset li {
-	display: block;
-	margin: 0;
-	padding: 0;
-	list-style: none;
-}
-
-.buttonset {
-	margin: 15px 0;
-}
-
-.button, .buttonset li a, input[type=submit] {
-	cursor: pointer;
-}
-
-.button, .buttonset li a, input[type=submit], input[type=text],
-		input[type=password], textarea, select, .multichoice_selector {
-	text-align: center;
-	display: inline-block;
-	padding: 0.4em 1em;
-	background: rgba(255,255,255,0.07);
-	border: 1px solid rgba(255,255,255,0.1);
-	color: #ddd;
-	border-radius: 5px;
-	text-decoration: none;
-	font-size: 100%;
-}
-
-input[type=text], input[type=password], textarea, select, .multichoice_selector {
-	text-align: left;
-}
-
 .ui-autocomplete, ui-front {
 	position:absolute;
 	cursor:default;
@@ -129,75 +24,12 @@ input[type=text], input[type=password], textarea, select, .multichoice_selector
 	}
 }
 
-select {
-	min-width: 200px;
+.bulletselector {
+	height: auto !important;
+	display: inline-block !important;
 }
 
-select:not([multiple]) {
-	background: linear-gradient(#444, #333);
-}
-
-.form-group {
-	padding: 8px 0;
-}
-
-.form-group label {
-	display: block;
-	vertical-align: top;
-	padding: 0 8px 8px 0;
-}
-
-.form-group input, .form-group textarea, .form-group .multichoice_selector {
-	display: block;
-	min-width: 100%;
-	max-width: 100%;
-}
-
-.box .form-group input, .box .form-group textarea, .form-group .multichoice_selector {
-	min-width: 95%;
-	max-width: 95%;
-}
-
-.form-group textarea {
-	min-height: 200px;
-}
-
-.button:hover, .buttonset li a:hover, input[type=submit]:hover {
-	background: rgba(255,255,255,0.13);
-	border: 1px solid rgba(255,255,255,0.17);
-	text-decoration: none;
-	color: #ddd;
-}
-
-.btn_green {
-	background: #363 !important;
-	border: 1px solid #473;
-}
-
-.btn_green:hover {
-	background: #474 !important;
-}
-
-.linedbuttonset a {
-	border: 1px solid #eee;
-	border-radius: 3px;
-	padding: 4px 10px;
-	margin: 0;
-	display: block;
-}
-
-.linedbuttonset {
-	display: block;
-	margin: 0;
-}
-
-.linedbuttonset li {
-	display: inline-block;
-	margin: 10px 10px 0 0;
-}
-
-
-.multichoice_selector input {
+.bulletselector input {
 	border: none;
 	border-radius: 0;
     -moz-border-radius: 0;
@@ -211,152 +43,14 @@ select:not([multiple]) {
 	white-space: nowrap;
 	background: transparent;
 }
-.multichoice_selector .tag {
-	background: #375D81;
-	border-radius: 3px;
-    -moz-border-radius: 3px;
-	color: #FFF;
+.bulletselector .badge {
 	float: left;
-	height: 15px;
-	padding: 0.1em 0.4em 0.5em;
+	padding: 0.4em 0.8em;
 	margin-right: 0.3em;
-	margin-bottom: 0.3em;
-    vertical-align: baseline;
-}
-.multichoice_selector .tag a {
-	color: #FFF;
-	cursor: pointer;
-}
-.multichoice_selector .tag a:hover {
-	color: #0099CC;
-	text-decoration: none;
 }
 
-
-/* Alerts */
-
-.alert {
-	padding: 10px;
-	position: relative;
-}
-
-.alert .alert_right, .alert > form {
-	display: inline-block;
-	margin: 0;
-	padding: 0;
-	position: absolute;
-	top: 0;
-	right: 0;
-	bottom: 0;
-}
-
-.alert .alert_right form {
-	height: 100%;
-}
-
-.alert form {
-	display: inline-block;
-	margin: 0;
-	padding: 0;
-}
-
-.alert input {
-	margin: 0;
-	height: 100%;
-	background: 0;
-	border: 0;
-	border-left: 1px solid rgba(255,255,255,0.12);
-	border-radius: 0;
-}
-
-.alert input:hover {
-	border: 0;
-	border-left: 1px solid rgba(255,255,255,0.2);
-}
-
-#alerts {
-	list-style: none;
-	position: fixed;
-	bottom: 15px;
-	left: 0;
-	right: 0;
-}
-
-#alerts .alert {
-	margin: 5px 0;
-	vertical-align: middle;
-}
-
-#alerts .close {
-	float: right;
-	color: white;
-}
-
-#alerts .close:hover {
-	color: #fff;
-}
-
-.alert-error, .button-danger {
-	background: #933 !important;
-	border: 1px solid #c44 !important;
-}
-
-.alert-warning {
-	background: #963;
-	border: 1px solid #c96;
-}
-
-.alert-success {
-	background: #161;
-	border: 1px solid #393;
-}
-
-table.fancyTable {
-	font-family: "Arial Black", Gadget, sans-serif;
-	border: 2px solid #000000;
-	background-color: #4A4A4A;
-	width: 100%;
-	text-align: center;
-	border-collapse: collapse;
-}
-table.fancyTable td, table.fancyTable th {
-	border: 1px solid #4A4A4A;
-	padding: 3px 2px;
-}
-table.fancyTable tbody td {
-	font-size: 13px;
-	color: #E6E6E6;
-}
-table.fancyTable tr:nth-child(even) {
-	background: #888888;
-}
-table.fancyTable thead {
-	background: #000000;
-	border-bottom: 3px solid #000000;
-}
-table.fancyTable thead th {
-	font-size: 15px;
-	font-weight: bold;
-	color: #E6E6E6;
-	text-align: center;
-	border-left: 2px solid #4A4A4A;
-}
-table.fancyTable thead th:first-child {
-	border-left: none;
-}
-
-table.fancyTable tfoot {
-	font-size: 12px;
-	font-weight: bold;
-	color: #E6E6E6;
-	background: #000000;
-	background: -moz-linear-gradient(top, #404040 0%, #191919 66%, #000000 100%);
-	background: -webkit-linear-gradient(top, #404040 0%, #191919 66%, #000000 100%);
-	background: linear-gradient(to bottom, #404040 0%, #191919 66%, #000000 100%);
-	border-top: 1px solid #4A4A4A;
-}
-table.fancyTable tfoot td {
-	font-size: 12px;
+.invalid-remaining {
+	display: none;
 }
 
 .t-mll tr td:not(:first-child) {
@@ -383,53 +77,48 @@ table.fancyTable tfoot td {
 	color: #b6f;
 }
 
-/*
-	Aside
-*/
-
-.asideright {
-	float: right;
-	margin: 0 0 0 15px;
-	max-width: 300px;
+.TRUSTED_MEMBER a, a.TRUSTED_MEMBER {
+	color: #2c2;
 }
 
-.outsidecontainer {
-	position: absolute;
-	right: 102%;
-	top: 0;
-	width: intrinsic;           /* Safari/WebKit uses a non-standard name */
-	width: -moz-max-content;    /* Firefox/Gecko */
-	width: -webkit-max-content; /* Chrome */
+.wiptopic a:not(.btn) {
+	color: #7ac;
 }
 
-@media (max-width: 1490px) {
-	.outsidecontainer {
-		display: none;
+.discardtopic {
+	text-decoration: line-through;
+	a:not(.btn) {
+		color: #7ac;
 	}
+	filter: brightness(0.5);
 }
 
-.flatlist, .flatlist li {
-	list-style: none;
-	padding: 0;
-	margin: 0;
+.editor-toolbar, .editor-toolbar.fullscreen {
+	margin-bottom: 0 !important;
+	background-color: #444 !important;
+	border: none !important;
+	border-radius: calc(0.25rem - 1px) calc(0.25rem - 1px) 0 0 !important;
+	border-bottom: 1px solid rgba(0, 0, 0, 0.125) !important;
 }
 
-.flatlist li {
-	display: block;
+.editor-toolbar button {
+	color: white;
 }
 
-.flatlist a {
-	display: block;
-	padding: 0.5em 20px;
-	color: #ddd;
-	font-weight: normal;
+.editor-toolbar button.active, .editor-toolbar button:hover {
+	background: #375a7f !important;
+	color: white !important;
 }
 
-.flatlist a:hover {
-	background: #444;
-	text-decoration: none;
+.editor-toolbar.fullscreen::before, .editor-toolbar.fullscreen::after {
+	display: none !important;
 }
 
-.table-topalign td {
-	vertical-align: top;
+// .CodeMirror {
+// 	background-color: #222 !important;
+// }
+
+.editor-preview-side, .editor-preview {
+	background-color: #222 !important;
+	color: white !important;
 }

app/scss/custom.scss

@@ -0,0 +1,63 @@
+@import "components.scss";
+@import "packages.scss";
+@import "packagegrid.scss";
+@import "comments.scss";
+
+.dropdown-menu {
+	margin-top: 0;
+}
+
+.dropdown:hover .dropdown-menu {
+   display: block;
+}
+
+.nav-link > img {
+	max-height: 1em;
+}
+
+#alerts {
+	display: block;
+	list-style: none;
+	position: fixed;
+	bottom: 0;
+	left:0;
+	right:0;
+	margin: 0;
+	padding:0;
+	z-index: 1000;
+}
+
+#alerts li {
+	list-style: none;
+}
+
+.jumbotron {
+	background-size: cover;
+	background-repeat: no-repeat;
+	background-position: center;
+
+	.btn-outline-secondary {
+		color: #eee;
+		border-color: #666;
+		background: rgba(102, 102, 102, 0.3);
+	}
+}
+
+.alert .btn {
+	text-decoration: none;
+}
+
+.card .table {
+	margin-bottom: 0;
+}
+
+.btn-download {
+	color: #fff;
+	background-color: #00b05c;
+	border-color: #00b05c;
+}
+
+.btn-download:focus, .btn-download.focus {
+	-webkit-box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
+	box-shadow: 0 0 0 0.2rem rgba(231, 76, 60, 0.5);
+}

app/scss/main.scss

@@ -1,5 +0,0 @@
-@import "page.scss";
-@import "components.scss";
-@import "nav.scss";
-@import "packages.scss";
-@import "packagegrid.scss";

app/scss/nav.scss

@@ -1,90 +0,0 @@
-nav {
-	margin: 0 auto 0 auto;
-	list-style: none;
-	background: #333;
-}
-
-nav .navbar-left {
-	float: left;
-}
-
-nav .navbar-left li {
-	float: left;
-}
-
-nav .navbar-right {
-	float: right;
-}
-
-nav ul {
-	margin: 0 auto 0 auto;
-	padding: 0;
-	list-style: none;
-}
-
-nav li {
-	margin: 0;
-	padding: 0;
-	list-style: none;
-	display: inline-block;
-}
-
-nav li a {
-	color: #ddd;
-	margin: 0;
-	padding: 1em 1em;
-	display: block;
-	border-left: 1px solid #444;
-}
-
-nav li a:not([href]) {
-	cursor: default;
-}
-
-nav ul li:last-child a {
-	border-right: 1px solid #444;
-}
-
-nav a:hover {
-	color: #eee;
-	background: #444;
-	text-decoration: none;
-}
-
-nav img {
-	height: 1em;
-}
-
-li.dropdown {
-	position: relative;
-	display: inline-block;
-}
-
-.dropdown-menu {
-	display: none;
-	position: absolute;
-	margin: 0;
-	padding: 0;
-	min-width:160px;
-	background: #333;
-	z-index: 1;
-	right: 0;
-	box-shadow: 0px 8px 16px 0px rgba(0,0,0,0.4);
-}
-
-.dropdown:hover ul {
-	display: block;
-}
-
-.dropdown li {
-	display: block;
-}
-
-.dropdown li a {
-	border: none;
-	border-top: 1px solid #444;
-}
-
-.dropdown li:last-child a {
-	border-bottom: 1px solid #444;
-}

app/scss/packagegrid.scss

@@ -1,26 +1,20 @@
-.packagegrid {
-	display: flex;
-	flex-wrap: wrap;
-	flex-direction: row;
-	flex-grow: 1;
-	flex-shrink: 1;
+.packagetile {
+	list-style: none;
 	padding: 0;
-	margin: 0 -7px;
+	margin: 0 7px 7px 0;
+	min-width: 250px;
 }
 
-.packagegrid li {
-	flex: 1;
-	display: block;
-	min-width: 300px;
-	min-height: 200px;
+li.d-flex {
+	list-style: none;
 	padding: 0;
-	margin: 7px;
+	margin: 0;
 }
 
-.packagegrid a {
+.packagetile a {
 	display: block;
 	padding-bottom: 66.66%;
-	border-radius: 7px;
+	border-radius: 3px;
 	position: relative;
 	overflow: hidden;
 	background-size: cover;
@@ -28,10 +22,6 @@
 	background-position: center;
 }
 
-.packagegrid a:hover {
-	// box-shadow: 0px 0px 16px 6px rgba(0,0,0,0.4);
-}
-
 .packagegridscrub {
 	position: absolute;
 	top: 50%;
@@ -52,6 +42,14 @@
 
 .packagegridinfo h3 {
 	color: white;
+	font-size: 120%;
+	font-weight: bold;
+}
+
+.packagegridinfo small {
+	color: #ddd;
+	font-size: 75%;
+	font-weight: bold;
 }
 
 .packagegridinfo p {
@@ -60,15 +58,15 @@
 	font-weight: normal;
 }
 
-.packagegrid a:hover .packagegridinfo {
+.packagetile a:hover .packagegridinfo {
 	top: 0;
 }
 
-.packagegrid a:hover p {
+.packagetile a:hover p {
 	display: block;
 }
 
-.packagegrid a:hover .packagegridscrub {
+.packagetile a:hover .packagegridscrub {
 	top: 0;
 	background: rgba(0,0,0,0.8);
 }

app/scss/packages.scss

@@ -37,34 +37,6 @@
 	left: 15px;
 }
 
-.sidebar_container {
-	display: block;
-	position: relative;
-	padding: 0;
-	margin: 0;
-}
-
-.sidebar_container .right, .sidebar_container .left{
-	position: absolute;
-	display: block;
-	top: 10px;
-	margin-top: 0;
-}
-
-.sidebar_container .right {
-	right: 0;
-	width: 280px;
-}
-
-.sidebar_container .left {
-	right: 295px;
-	left: 0;
-}
-
-.sidebar_container .right > *:first-child, .sidebar_container .left > *:first-child {
-	margin-top: 0;
-}
-
 .package-short-large {
 	font-size: 120%;
 }

app/scss/page.scss

@@ -1,56 +0,0 @@
-html, body {
-	font-family: "Arial", sans-serif;
-	background: #222;
-	color: #ddd;
-	padding: 0;
-	margin: 0;
-}
-
-.container, main, #alerts, footer {
-	width: 90%;
-	max-width: 1024px;
-	margin: auto;
-	padding: 0;
-	display: block;
-}
-
-main {
-	padding: 7px 0;
-	position: relative;
-	box-sizing: border-box;
-}
-
-.clearboth {
-	clear: both;
-}
-
-header h1, header p, header form {
-	padding: 0 5px;
-	margin-left: 0;
-}
-
-header {
-	padding: 30px;
-	background: #258;
-}
-
-header p {
-	max-width: 400px;
-}
-
-header input {
-	margin: 3px;
-}
-
-footer {
-	color: #999;
-	padding: 30px 0;
-}
-
-footer a {
-	color: #aaa;
-}
-
-footer a:hover {
-	color: #bbb;
-}

app/tasks/__init__.py

@@ -16,8 +16,9 @@
 
 
 import flask
-from flask.ext.sqlalchemy import SQLAlchemy
+from flask_sqlalchemy import SQLAlchemy
 from celery import Celery
+from celery.schedules import crontab
 from app import app
 from app.models import *
 
@@ -64,4 +65,16 @@ def make_celery(app):
 
 celery = make_celery(app)
 
-from . import importtasks, forumtasks, emails
+CELERYBEAT_SCHEDULE = {
+	'topic_list_import': {
+		'task': 'app.tasks.forumtasks.importTopicList',
+		'schedule': crontab(minute=1, hour=1),
+	},
+	'package_score_update': {
+		'task': 'app.tasks.pkgtasks.updatePackageScores',
+		'schedule': crontab(minute=10, hour=1),
+	}
+}
+celery.conf.beat_schedule = CELERYBEAT_SCHEDULE
+
+from . import importtasks, forumtasks, emails, pkgtasks

app/tasks/emails.py

@@ -15,14 +15,36 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-from flask import *
+from flask import render_template, url_for
 from flask_mail import Message
 from app import mail
 from app.tasks import celery
 
 @celery.task()
 def sendVerifyEmail(newEmail, token):
-    msg = Message("Verify email address", recipients=[newEmail])
-    msg.body = "This is a verification email!"
-    msg.html = render_template("emails/verify.html", token=token)
-    mail.send(msg)
+	print("Sending verify email!")
+	msg = Message("Verify email address", recipients=[newEmail])
+
+	msg.body = """
+			This email has been sent to you because someone (hopefully you)
+			has entered your email address as a user's email.
+
+			If it wasn't you, then just delete this email.
+
+			If this was you, then please click this link to verify the address:
+
+			{}
+		""".format(url_for('users.verify_email', token=token, _external=True))
+
+	msg.html = render_template("emails/verify.html", token=token)
+	mail.send(msg)
+
+@celery.task()
+def sendEmailRaw(to, subject, text, html):
+	from flask_mail import Message
+	msg = Message(subject, recipients=to)
+
+	msg.body = text or html
+	html = html or text
+	msg.html = render_template("emails/base.html", subject=subject, content=html)
+	mail.send(msg)

app/tasks/forumtasks.py

@@ -15,17 +15,18 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import flask, json
-from flask.ext.sqlalchemy import SQLAlchemy
+import flask, json, re
+from flask_sqlalchemy import SQLAlchemy
 from app import app
 from app.models import *
 from app.tasks import celery
-from .phpbbparser import getProfile
+from .phpbbparser import getProfile, getTopicsFromForum
 import urllib.request
 from urllib.parse import urlparse, quote_plus
 
 @celery.task()
-def checkForumAccount(username, token=None):
+def checkForumAccount(username, forceNoSave=False):
+	print("Checking " + username)
 	try:
 		profile = getProfile("https://forum.minetest.net", username)
 	except OSError:
@@ -47,31 +48,127 @@ def checkForumAccount(username, token=None):
 		user.github_username = github_username
 		needsSaving = True
 
+	pic = profile.avatar
+	if pic and "http" in pic:
+		pic = None
+
+	needsSaving = needsSaving or pic != user.profile_pic
+	if pic:
+		user.profile_pic = "https://forum.minetest.net/" + pic
+	else:
+		user.profile_pic = None
+
 	# Save
-	if needsSaving:
+	if needsSaving and not forceNoSave:
 		db.session.commit()
 
+	return needsSaving
+
 @celery.task()
-def importUsersFromModList():
-	contents = urllib.request.urlopen("http://krock-works.16mb.com/MTstuff/modList.php").read().decode("utf-8")
-	list = json.loads(contents)
-	found = {}
-	imported = []
+def checkAllForumAccounts(forceNoSave=False):
+	needsSaving = False
+	query = User.query.filter(User.forums_username.isnot(None))
+	for user in query.all():
+		needsSaving = checkForumAccount(user.username) or needsSaving
 
-	for user in User.query.all():
-		found[user.username] = True
-		if user.forums_username is not None:
-			found[user.forums_username] = True
+	if needsSaving and not forceNoSave:
+		db.session.commit()
 
-	for x in list:
-		author = x.get("author")
-		if author is not None and not author in found:
-			user = User(author)
-			user.forums_username = author
-			imported.append(author)
-			found[author] = True
-			db.session.add(user)
+	return needsSaving
+
+
+regex_tag    = re.compile(r"\[([a-z0-9_]+)\]")
+BANNED_NAMES = ["mod", "game", "old", "outdated", "wip", "api", "beta", "alpha", "git"]
+def getNameFromTaglist(taglist):
+	for tag in reversed(regex_tag.findall(taglist)):
+		if len(tag) < 30 and not tag in BANNED_NAMES and \
+				not re.match(r"^[a-z]?[0-9]+$", tag):
+			return tag
+
+	return None
+
+regex_title = re.compile(r"^((?:\[[^\]]+\] *)*)([^\[]+) *((?:\[[^\]]+\] *)*)[^\[]*$")
+def parseTitle(title):
+	m = regex_title.match(title)
+	if m is None:
+		print("Invalid title format: " + title)
+		return title, getNameFromTaglist(title)
+	else:
+		return m.group(2).strip(), getNameFromTaglist(m.group(3))
+
+def getLinksFromModSearch():
+	links = {}
+
+	try:
+		contents = urllib.request.urlopen("https://krock-works.uk.to/minetest/modList.php").read().decode("utf-8")
+		for x in json.loads(contents):
+			try:
+				link = x.get("link")
+				if link is not None:
+					links[int(x["topicId"])] = link
+			except ValueError:
+				pass
+
+	except urllib.error.URLError:
+		print("Unable to open krocks mod search!")
+		return links
+
+	return links
+
+@celery.task()
+def importTopicList():
+	links_by_id = getLinksFromModSearch()
+
+	info_by_id = {}
+	getTopicsFromForum(11, out=info_by_id, extra={ 'type': PackageType.MOD,  'wip': False })
+	getTopicsFromForum(9,  out=info_by_id, extra={ 'type': PackageType.MOD,  'wip': True  })
+	getTopicsFromForum(15, out=info_by_id, extra={ 'type': PackageType.GAME, 'wip': False })
+	getTopicsFromForum(50, out=info_by_id, extra={ 'type': PackageType.GAME, 'wip': True  })
+
+	# Caches
+	username_to_user = {}
+	topics_by_id     = {}
+	for topic in ForumTopic.query.all():
+		topics_by_id[topic.topic_id] = topic
+
+	# Create or update
+	for info in info_by_id.values():
+		id = int(info["id"])
+
+		# Get author
+		username = info["author"]
+		user = username_to_user.get(username)
+		if user is None:
+			user = User.query.filter_by(forums_username=username).first()
+			if user is None:
+				print(username + " not found!")
+				user = User(username)
+				user.forums_username = username
+				db.session.add(user)
+			username_to_user[username] = user
+
+		# Get / add row
+		topic = topics_by_id.get(id)
+		if topic is None:
+			topic = ForumTopic()
+			db.session.add(topic)
+
+		# Parse title
+		title, name = parseTitle(info["title"])
+
+		# Get link
+		link = links_by_id.get(id)
+
+		# Fill row
+		topic.topic_id   = int(id)
+		topic.author     = user
+		topic.type       = info["type"]
+		topic.title      = title
+		topic.name       = name
+		topic.link       = link
+		topic.wip        = info["wip"]
+		topic.posts      = int(info["posts"])
+		topic.views      = int(info["views"])
+		topic.created_at = info["date"]
 
 	db.session.commit()
-	for author in found:
-		checkForumAccount.delay(author, None)

app/tasks/importtasks.py

@@ -15,18 +15,28 @@
 # along with this program.  If not, see <https://www.gnu.org/licenses/>.
 
 
-import flask, json, os
-from flask.ext.sqlalchemy import SQLAlchemy
+import flask, json, os, git, tempfile, shutil, gitdb
+from git import GitCommandError
+from git_archive_all import GitArchiver
+from flask_sqlalchemy import SQLAlchemy
 from urllib.error import HTTPError
 import urllib.request
-from urllib.parse import urlparse, quote_plus
+from urllib.parse import urlparse, quote_plus, urlsplit
+from zipfile import ZipFile
+
 from app import app
 from app.models import *
 from app.tasks import celery, TaskError
 from app.utils import randomString
+from .minetestcheck import build_tree, MinetestCheckError, ContentType
+from .minetestcheck.config import parse_conf
 
 class GithubURLMaker:
 	def __init__(self, url):
+		self.baseUrl = None
+		self.user = None
+		self.repo = None
+
 		# Rewrite path
 		import re
 		m = re.search("^\/([^\/]+)\/([^\/]+)\/?$", url.path)
@@ -46,18 +56,12 @@ class GithubURLMaker:
 	def getRepoURL(self):
 		return "https://github.com/{}/{}".format(self.user, self.repo)
 
-	def getIssueTrackerURL(self):
-		return "https://github.com/{}/{}/issues/".format(self.user, self.repo)
+	def getScreenshotURL(self):
+		return self.baseUrl + "/screenshot.png"
 
 	def getModConfURL(self):
 		return self.baseUrl + "/mod.conf"
 
-	def getDescURL(self):
-		return self.baseUrl + "/description.txt"
-
-	def getScreenshotURL(self):
-		return self.baseUrl + "/screenshot.png"
-
 	def getCommitsURL(self, branch):
 		return "https://api.github.com/repos/{}/{}/commits?sha={}" \
 				.format(self.user, self.repo, urllib.parse.quote_plus(branch))
@@ -66,7 +70,6 @@ class GithubURLMaker:
 		return "https://github.com/{}/{}/archive/{}.zip" \
 				.format(self.user, self.repo, commit)
 
-
 krock_list_cache = None
 krock_list_cache_by_name = None
 def getKrockList():
@@ -74,7 +77,7 @@ def getKrockList():
 	global krock_list_cache_by_name
 
 	if krock_list_cache is None:
-		contents = urllib.request.urlopen("http://krock-works.16mb.com/MTstuff/modList.php").read().decode("utf-8")
+		contents = urllib.request.urlopen("https://krock-works.uk.to/minetest/modList.php").read().decode("utf-8")
 		list = json.loads(contents)
 
 		def h(x):
@@ -94,9 +97,9 @@ def getKrockList():
 			return {
 				"title":   x["title"],
 				"author":  x["author"],
-				"name":    x["name"],
+				"name":	x["name"],
 				"topicId": x["topicId"],
-				"link":    x["link"],
+				"link":	x["link"],
 			}
 
 		krock_list_cache = [g(x) for x in list if h(x)]
@@ -127,106 +130,178 @@ def findModInfo(author, name, link):
 
 	return None
 
+def generateGitURL(urlstr):
+	scheme, netloc, path, query, frag = urlsplit(urlstr)
 
-def parseConf(string):
-	retval = {}
-	for line in string.split("\n"):
-		idx = line.find("=")
-		if idx > 0:
-			key   = line[:idx].strip()
-			value = line[idx+1:].strip()
-			retval[key] = value
+	return "http://:@" + netloc + path + query
 
-	return retval
 
+def getTempDir():
+	return os.path.join(tempfile.gettempdir(), randomString(10))
+
+
+# Clones a repo from an unvalidated URL.
+# Returns a tuple of path and repo on sucess.
+# Throws `TaskError` on failure.
+# Caller is responsible for deleting returned directory.
+def cloneRepo(urlstr, ref=None, recursive=False):
+	gitDir = getTempDir()
+
+	err = None
+	try:
+		gitUrl = generateGitURL(urlstr)
+		print("Cloning from " + gitUrl)
+
+		if ref is None:
+			repo = git.Repo.clone_from(gitUrl, gitDir, \
+					progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15)
+		else:
+			repo = git.Repo.clone_from(gitUrl, gitDir, \
+					progress=None, env=None, depth=1, recursive=recursive, kill_after_timeout=15, b=ref)
+
+		return gitDir, repo
+
+	except GitCommandError as e:
+		# This is needed to stop the backtrace being weird
+		err = e.stderr
+
+	except gitdb.exc.BadName as e:
+		err = "Unable to find the reference " + (ref or "?") + "\n" + e.stderr
+
+	raise TaskError(err.replace("stderr: ", "") \
+			.replace("Cloning into '" + gitDir + "'...", "") \
+			.strip())
 
 @celery.task()
 def getMeta(urlstr, author):
-	url = urlparse(urlstr)
-
-	urlmaker = None
-	if url.netloc == "github.com":
-		urlmaker = GithubURLMaker(url)
-	else:
-		raise TaskError("Unsupported repo")
-
-	if not urlmaker.isValid():
-		raise TaskError("Error! Url maker not valid")
-
-	result = {}
-
-	result["repo"] = urlmaker.getRepoURL()
-	result["issueTracker"] = urlmaker.getIssueTrackerURL()
+	gitDir, _ = cloneRepo(urlstr, recursive=True)
 
 	try:
-		contents = urllib.request.urlopen(urlmaker.getModConfURL()).read().decode("utf-8")
-		conf = parseConf(contents)
-		for key in ["name", "description", "title"]:
-			try:
-				result[key] = conf[key]
-			except KeyError:
-				pass
-	except HTTPError:
-		print("mod.conf does not exist")
+		tree = build_tree(gitDir, author=author, repo=urlstr)
+	except MinetestCheckError as err:
+		raise TaskError(str(err))
 
-	if "name" in result:
-		result["title"] = result["name"].replace("_", " ").title()
+	shutil.rmtree(gitDir)
 
-	if not "description" in result:
-		try:
-			contents = urllib.request.urlopen(urlmaker.getDescURL()).read().decode("utf-8")
-			result["description"] = contents.strip()
-		except HTTPError:
-			print("description.txt does not exist!")
+	result = {}
+	result["name"] = tree.name
+	result["provides"] = tree.fold("name")
+	result["type"] = tree.type.name
 
-	if "description" in result:
-		desc = result["description"]
-		idx = desc.find(".") + 1
-		cutIdx = min(len(desc), 200 if idx < 5 else idx)
-		result["short_description"] = desc[:cutIdx]
+	for key in ["depends", "optional_depends"]:
+		result[key] = tree.fold("meta", key)
 
-	info = findModInfo(author, result.get("name"), result["repo"])
-	if info is not None:
-		result["forumId"] = info.get("topicId")
+	for key in ["title", "repo", "issueTracker", "forumId", "description", "short_description"]:
+		result[key] = tree.get(key)
+
+	for mod in result["provides"]:
+		result["depends"].discard(mod)
+		result["optional_depends"].discard(mod)
+
+	for key, value in result.items():
+		if isinstance(value, set):
+			result[key] = list(value)
 
 	return result
 
 
+def makeVCSReleaseFromGithub(id, branch, release, url):
+	urlmaker = GithubURLMaker(url)
+	if not urlmaker.isValid():
+		raise TaskError("Invalid github repo URL")
+
+	commitsURL = urlmaker.getCommitsURL(branch)
+	try:
+		contents = urllib.request.urlopen(commitsURL).read().decode("utf-8")
+		commits = json.loads(contents)
+	except HTTPError:
+		raise TaskError("Unable to get commits for Github repository. Either the repository or reference doesn't exist.")
+
+	if len(commits) == 0 or not "sha" in commits[0]:
+		raise TaskError("No commits found")
+
+	release.url          = urlmaker.getCommitDownload(commits[0]["sha"])
+	release.task_id     = None
+	release.commit_hash = commits[0]["sha"]
+	release.approve(release.package.author)
+	db.session.commit()
+
+	return release.url
+
+
+@celery.task(bind=True)
+def checkZipRelease(self, id, path):
+	release = PackageRelease.query.get(id)
+	if release is None:
+		raise TaskError("No such release!")
+	elif release.package is None:
+		raise TaskError("No package attached to release")
+
+	temp = getTempDir()
+	try:
+		with ZipFile(path, 'r') as zip_ref:
+			zip_ref.extractall(temp)
+
+		try:
+			tree = build_tree(temp, expected_type=ContentType[release.package.type.name], \
+				author=release.package.author.username, name=release.package.name)
+		except MinetestCheckError as err:
+			if "Fails validation" not in release.title:
+				release.title += " (Fails validation)"
+
+			release.task_id = self.request.id
+			release.approved = False
+			db.session.commit()
+
+			raise TaskError(str(err))
+
+		release.task_id = None
+		release.approve(release.package.author)
+		db.session.commit()
+
+	finally:
+		shutil.rmtree(temp)
+
+
 @celery.task()
 def makeVCSRelease(id, branch):
 	release = PackageRelease.query.get(id)
-
 	if release is None:
 		raise TaskError("No such release!")
-
-	if release.package is None:
+	elif release.package is None:
 		raise TaskError("No package attached to release")
 
-	url = urlparse(release.package.repo)
+	# url = urlparse(release.package.repo)
+	# if url.netloc == "github.com":
+	# 	return makeVCSReleaseFromGithub(id, branch, release, url)
 
-	urlmaker = None
-	if url.netloc == "github.com":
-		urlmaker = GithubURLMaker(url)
-	else:
-		raise TaskError("Unsupported repo")
+	gitDir, repo = cloneRepo(release.package.repo, ref=branch, recursive=True)
 
-	if not urlmaker.isValid():
-		raise TaskError("Invalid github repo URL")
+	try:
+		tree = build_tree(gitDir, expected_type=ContentType[release.package.type.name], \
+			author=release.package.author.username, name=release.package.name)
+	except MinetestCheckError as err:
+		raise TaskError(str(err))
 
-	commitsURL = urlmaker.getCommitsURL(branch)
-	contents = urllib.request.urlopen(commitsURL).read().decode("utf-8")
-	commits = json.loads(contents)
+	try:
+		filename = randomString(10) + ".zip"
+		destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
 
-	if len(commits) == 0 or not "sha" in commits[0]:
-		raise TaskError("No commits found")
+		assert(not os.path.isfile(destPath))
+		archiver = GitArchiver(force_sub=True, main_repo_abspath=gitDir)
+		archiver.create(destPath)
+		assert(os.path.isfile(destPath))
 
-	release.url = urlmaker.getCommitDownload(commits[0]["sha"])
-	print(release.url)
-	release.task_id = None
-	db.session.commit()
-
-	return release.url
+		release.url         = "/uploads/" + filename
+		release.task_id     = None
+		release.commit_hash = repo.head.object.hexsha
+		release.approve(release.package.author)
+		print(release.url)
+		db.session.commit()
 
+		return release.url
+	finally:
+		shutil.rmtree(gitDir)
 
 @celery.task()
 def importRepoScreenshot(id):
@@ -235,32 +310,122 @@ def importRepoScreenshot(id):
 		raise Exception("Unexpected none package")
 
 	# Get URL Maker
+	try:
+		gitDir, _ = cloneRepo(package.repo)
+	except TaskError as e:
+		# ignore download errors
+		print(e)
+		return None
+
+	# Find and import screenshot
+	try:
+		for ext in ["png", "jpg", "jpeg"]:
+			sourcePath = gitDir + "/screenshot." + ext
+			if os.path.isfile(sourcePath):
+				filename = randomString(10) + "." + ext
+				destPath = os.path.join(app.config["UPLOAD_DIR"], filename)
+				shutil.copyfile(sourcePath, destPath)
+
+				ss = PackageScreenshot()
+				ss.approved = True
+				ss.package = package
+				ss.title   = "screenshot.png"
+				ss.url	 = "/uploads/" + filename
+				db.session.add(ss)
+				db.session.commit()
+
+				return "/uploads/" + filename
+	finally:
+		shutil.rmtree(gitDir)
+
+	print("screenshot.png does not exist")
+	return None
+
+
+
+def getDepends(package):
 	url = urlparse(package.repo)
 	urlmaker = None
 	if url.netloc == "github.com":
 		urlmaker = GithubURLMaker(url)
 	else:
-		raise TaskError("Unsupported repo")
+		return {}
 
+	result = {}
 	if not urlmaker.isValid():
-		raise TaskError("Error! Url maker not valid")
+		return {}
 
+	#
+	# Try getting depends on mod.conf
+	#
 	try:
-		filename = randomString(10) + ".png"
-		imagePath = os.path.join("app/public/uploads", filename)
-		print(imagePath)
-		urllib.request.urlretrieve(urlmaker.getScreenshotURL(), imagePath)
+		contents = urllib.request.urlopen(urlmaker.getModConfURL()).read().decode("utf-8")
+		conf = parse_conf(contents)
+		for key in ["depends", "optional_depends"]:
+			try:
+				result[key] = conf[key]
+			except KeyError:
+				pass
 
-		ss = PackageScreenshot()
-		ss.approved = True
-		ss.package = package
-		ss.title   = "screenshot.png"
-		ss.url     = "/uploads/" + filename
-		db.session.add(ss)
-		db.session.commit()
-
-		return "/uploads/" + filename
 	except HTTPError:
-		print("screenshot.png does not exist")
+		print("mod.conf does not exist")
 
-	return None
+	if "depends" in result or "optional_depends" in result:
+		return result
+
+
+	#
+	# Try depends.txt
+	#
+	import re
+	pattern = re.compile("^([a-z0-9_]+)\??$")
+	try:
+		contents = urllib.request.urlopen(urlmaker.getDependsURL()).read().decode("utf-8")
+		soft = []
+		hard = []
+		for line in contents.split("\n"):
+			line = line.strip()
+			if pattern.match(line):
+				if line[len(line) - 1] == "?":
+					soft.append( line[:-1])
+				else:
+					hard.append(line)
+
+		result["depends"] = ",".join(hard)
+		result["optional_depends"] = ",".join(soft)
+	except HTTPError:
+		print("depends.txt does not exist")
+
+	return result
+
+
+def importDependencies(package, mpackage_cache):
+	if Dependency.query.filter_by(depender=package).count() != 0:
+		return
+
+	result = getDepends(package)
+
+	if "depends" in result:
+		deps = Dependency.SpecToList(package, result["depends"], mpackage_cache)
+		print("{} hard: {}".format(len(deps), result["depends"]))
+		for dep in deps:
+			dep.optional = False
+			db.session.add(dep)
+
+	if "optional_depends" in result:
+		deps = Dependency.SpecToList(package, result["optional_depends"], mpackage_cache)
+		print("{} soft: {}".format(len(deps), result["optional_depends"]))
+		for dep in deps:
+			dep.optional = True
+			db.session.add(dep)
+
+@celery.task()
+def importAllDependencies():
+	Dependency.query.delete()
+	mpackage_cache = {}
+	packages = Package.query.filter_by(type=PackageType.MOD).all()
+	for i, p in enumerate(packages):
+		print("============= {} ({}/{}) =============".format(p.name, i, len(packages)))
+		importDependencies(p, mpackage_cache)
+
+	db.session.commit()

app/tasks/minetestcheck/__init__.py

@@ -0,0 +1,48 @@
+from enum import Enum
+
+class MinetestCheckError(Exception):
+	def __init__(self, value):
+		self.value = value
+	def __str__(self):
+		return repr("Error validating package: " + self.value)
+
+class ContentType(Enum):
+	UNKNOWN = "unknown"
+	MOD = "mod"
+	MODPACK = "modpack"
+	GAME = "game"
+	TXP = "texture pack"
+
+	def isModLike(self):
+		return self == ContentType.MOD or self == ContentType.MODPACK
+
+	def validate_same(self, other):
+		"""
+		Whether or not `other` is an acceptable type for this
+		"""
+		assert(other)
+
+		if self == ContentType.MOD:
+			if not other.isModLike():
+				raise MinetestCheckError("expected a mod or modpack, found " + other.value)
+
+		elif self == ContentType.TXP:
+			if other != ContentType.UNKNOWN and other != ContentType.TXP:
+				raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
+
+		elif other != self:
+			raise MinetestCheckError("expected a " + self.value + ", found a " + other.value)
+
+
+from .tree import PackageTreeNode, get_base_dir
+
+def build_tree(path, expected_type=None, author=None, repo=None, name=None):
+	path = get_base_dir(path)
+
+	root = PackageTreeNode(path, "/", author=author, repo=repo, name=name)
+	assert(root)
+
+	if expected_type:
+		expected_type.validate_same(root.type)
+
+	return root

app/tasks/minetestcheck/config.py

@@ -0,0 +1,10 @@
+def parse_conf(string):
+	retval = {}
+	for line in string.split("\n"):
+		idx = line.find("=")
+		if idx > 0:
+			key   = line[:idx].strip()
+			value = line[idx+1:].strip()
+			retval[key] = value
+
+	return retval

app/tasks/minetestcheck/tree.py

@@ -0,0 +1,162 @@
+import os
+from . import MinetestCheckError, ContentType
+from .config import parse_conf
+
+def get_base_dir(path):
+	if not os.path.isdir(path):
+		raise IOError("Expected dir")
+
+	root, subdirs, files = next(os.walk(path))
+	if len(subdirs) == 1 and len(files) == 0:
+		return get_base_dir(path + "/" + subdirs[0])
+	else:
+		return path
+
+
+def detect_type(path):
+	if os.path.isfile(path + "/game.conf"):
+		return ContentType.GAME
+	elif os.path.isfile(path + "/init.lua"):
+		return ContentType.MOD
+	elif os.path.isfile(path + "/modpack.txt") or \
+			os.path.isfile(path + "/modpack.conf"):
+		return ContentType.MODPACK
+	elif os.path.isdir(path + "/mods"):
+		return ContentType.GAME
+	elif os.path.isfile(path + "/texture_pack.conf"):
+		return ContentType.TXP
+	else:
+		return ContentType.UNKNOWN
+
+
+class PackageTreeNode:
+	def __init__(self, baseDir, relative, author=None, repo=None, name=None):
+		print(baseDir)
+		self.baseDir  = baseDir
+		self.relative = relative
+		self.author   = author
+		self.name	 = name
+		self.repo	 = repo
+		self.meta	 = None
+		self.children = []
+
+		# Detect type
+		self.type = detect_type(baseDir)
+		self.read_meta()
+
+		if self.type == ContentType.GAME:
+			if not os.path.isdir(baseDir + "/mods"):
+				raise MinetestCheckError(("game at {} does not have a mods/ folder").format(self.relative))
+			self.add_children_from_mod_dir(baseDir + "/mods")
+		elif self.type == ContentType.MODPACK:
+			self.add_children_from_mod_dir(baseDir)
+
+
+	def read_meta(self):
+		result = {}
+
+		# .conf file
+		try:
+			with open(self.baseDir + "/mod.conf", "r") as myfile:
+				conf = parse_conf(myfile.read())
+				for key in ["name", "description", "title", "depends", "optional_depends"]:
+					try:
+						result[key] = conf[key]
+					except KeyError:
+						pass
+		except IOError:
+			pass
+
+		# description.txt
+		if not "description" in result:
+			try:
+				with open(self.baseDir + "/description.txt", "r") as myfile:
+					result["description"] = myfile.read()
+			except IOError:
+				pass
+
+		# depends.txt
+		import re
+		pattern = re.compile("^([a-z0-9_]+)\??$")
+		if not "depends" in result and not "optional_depends" in result:
+			try:
+				with open(self.baseDir + "/depends.txt", "r") as myfile:
+					contents = myfile.read()
+					soft = []
+					hard = []
+					for line in contents.split("\n"):
+						line = line.strip()
+						if pattern.match(line):
+							if line[len(line) - 1] == "?":
+								soft.append( line[:-1])
+							else:
+								hard.append(line)
+
+					result["depends"] = hard
+					result["optional_depends"] = soft
+
+			except IOError:
+				pass
+
+		else:
+			if "depends" in result:
+				result["depends"] = [x.strip() for x in result["depends"].split(",")]
+			if "optional_depends" in result:
+				result["optional_depends"] = [x.strip() for x in result["optional_depends"].split(",")]
+
+
+		# Calculate Title
+		if "name" in result and not "title" in result:
+			result["title"] = result["name"].replace("_", " ").title()
+
+		# Calculate short description
+		if "description" in result:
+			desc = result["description"]
+			idx = desc.find(".") + 1
+			cutIdx = min(len(desc), 200 if idx < 5 else idx)
+			result["short_description"] = desc[:cutIdx]
+
+		if "name" in result:
+			self.name = result["name"]
+			del result["name"]
+
+		self.meta = result
+
+	def add_children_from_mod_dir(self, dir):
+		for entry in next(os.walk(dir))[1]:
+			path = os.path.join(dir, entry)
+			if not entry.startswith('.') and os.path.isdir(path):
+				child = PackageTreeNode(path, self.relative + entry + "/", name=entry)
+				if not child.type.isModLike():
+					raise MinetestCheckError(("Expecting mod or modpack, found {} at {} inside {}") \
+							.format(child.type.value, child.relative, self.type.value))
+
+				self.children.append(child)
+
+
+	def fold(self, attr, key=None, acc=None):
+		if acc is None:
+			acc = set()
+
+		if self.meta is None:
+			return acc
+
+		at = getattr(self, attr)
+		value = at if key is None else at.get(key)
+
+		if isinstance(value, list):
+			acc |= set(value)
+		elif value is not None:
+			acc.add(value)
+
+		for child in self.children:
+			child.fold(attr, key, acc)
+
+		return acc
+
+	def get(self, key):
+		return self.meta.get(key)
+
+	def validate(self):
+		for child in self.children:
+			child.validate()

app/tasks/phpbbparser.py

@@ -5,6 +5,7 @@
 import urllib, socket
 from bs4 import *
 from urllib.parse import urljoin
+from datetime import datetime
 import urllib.request
 import os.path
 import time, re
@@ -14,8 +15,9 @@ def urlEncodeNonAscii(b):
 
 class Profile:
 	def __init__(self, username):
-		self.username = username
-		self.signature = ""
+		self.username   = username
+		self.signature  = ""
+		self.avatar     = None
 		self.properties = {}
 
 	def set(self, key, value):
@@ -32,6 +34,11 @@ def __extract_properties(profile, soup):
 	if el is None:
 		return None
 
+	res1 = el.find_all("dl")
+	imgs = res1[0].find_all("img")
+	if len(imgs) == 1:
+		profile.avatar = imgs[0]["src"]
+
 	res = el.find_all("dl", class_ = "left-box details")
 	if len(res) != 1:
 		return None
@@ -77,3 +84,72 @@ def getProfile(url, username):
 		__extract_properties(profile, soup)
 
 		return profile
+
+
+regex_id = re.compile(r"^.*t=([0-9]+).*$")
+
+def parseForumListPage(id, page, out, extra=None):
+	num_per_page = 30
+	start = page*num_per_page+1
+	print(" - Fetching page {} (topics {}-{})".format(page, start, start+num_per_page))
+
+	url = "https://forum.minetest.net/viewforum.php?f=" + str(id) + "&start=" + str(start)
+	r = urllib.request.urlopen(url).read().decode("utf-8")
+	soup = BeautifulSoup(r, "html.parser")
+
+	for row in soup.find_all("li", class_="row"):
+		classes = row.get("class")
+		if "sticky" in classes or "announce" in classes or "global-announce" in classes:
+			continue
+
+		topic = row.find("dl")
+
+		# Link info
+		link   = topic.find(class_="topictitle")
+		id	   = regex_id.match(link.get("href")).group(1)
+		title  = link.find(text=True)
+
+		# Date
+		left   = topic.find("dt")
+		date   = left.get_text().split("»")[1].strip()
+		date   = datetime.strptime(date, "%a %b %d, %Y %H:%M")
+		author = left.find_all("a")[-1].get_text().strip()
+
+		# Get counts
+		posts  = topic.find(class_="posts").find(text=True)
+		views  = topic.find(class_="views").find(text=True)
+
+		if id in out:
+			print("   - got {} again, title: {}".format(id, title))
+			assert title == out[id]['title']
+			return False
+
+		row = {
+			"id"    : id,
+			"title" : title,
+			"author": author,
+			"posts" : posts,
+			"views" : views,
+			"date"  : date
+		}
+
+		if extra is not None:
+			for key, value in extra.items():
+				row[key] = value
+
+		out[id] = row
+
+	return True
+
+def getTopicsFromForum(id, out={}, extra=None):
+	print("Fetching all topics from forum {}".format(id))
+	page = 0
+	while parseForumListPage(id, page, out, extra):
+		page = page + 1
+
+	return out
+
+def dumpTitlesToFile(topics, path):
+	with open(path, "w") as out_file:
+		for topic in topics.values():
+			out_file.write(topic["title"] + "\n")

app/tasks/pkgtasks.py

@@ -0,0 +1,23 @@
+# Content DB
+# Copyright (C) 2018  rubenwardy
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+
+from app.models import Package
+from app.tasks import celery
+
+@celery.task()
+def updatePackageScores():
+	Package.query.update({ "score": Package.score * 0.8 })

app/template_filters.py

@@ -0,0 +1,22 @@
+from . import app
+from urllib.parse import urlparse
+
+@app.context_processor
+def inject_debug():
+    return dict(debug=app.debug)
+
+@app.template_filter()
+def throw(err):
+	raise Exception(err)
+
+@app.template_filter()
+def domain(url):
+	return urlparse(url).netloc
+
+@app.template_filter()
+def date(value):
+    return value.strftime("%Y-%m-%d")
+
+@app.template_filter()
+def datetime(value):
+    return value.strftime("%Y-%m-%d %H:%M") + " UTC"

app/templates/admin/licenses/edit.html

@@ -0,0 +1,25 @@
+{% extends "base.html" %}
+
+{% block title %}
+	{% if license %}
+		Edit {{ license.name }}
+	{% else %}
+		New license
+	{% endif %}
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.license_list') }}">Back to list</a> |
+		<a href="{{ url_for('admin.create_edit_license') }}">New License</a>
+	</p>
+
+	{% from "macros/forms.html" import render_field, render_submit_field %}
+	<form method="POST" action="" enctype="multipart/form-data">
+		{{ form.hidden_tag() }}
+
+		{{ render_field(form.name) }}
+		{{ render_field(form.is_foss) }}
+		{{ render_submit_field(form.submit) }}
+	</form>
+{% endblock %}

app/templates/admin/licenses/list.html

@@ -0,0 +1,16 @@
+{% extends "base.html" %}
+
+{% block title %}
+Licenses
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.create_edit_license') }}">New License</a>
+	</p>
+	<ul>
+		{% for l in licenses %}
+			<li><a href="{{ url_for('admin.create_edit_license', name=l.name) }}">{{ l.name }}</a> [{{ l.is_foss and "Free" or "Non-free"}}]</li>
+		{% endfor %}
+	</ul>
+{% endblock %}

app/templates/admin/list.html

@@ -6,27 +6,37 @@
 
 {% block content %}
 	<ul>
-		<li><a href="{{ url_for('user_list_page') }}">User list</a></li>
-		<li><a href="{{ url_for('switch_user_page') }}">Sign in as another user</a></li>
+		<li><a href="{{ url_for('users.list_all') }}">User list</a></li>
+		<li><a href="{{ url_for('admin.tag_list') }}">Tag Editor</a></li>
+		<li><a href="{{ url_for('admin.license_list') }}">License Editor</a></li>
+		<li><a href="{{ url_for('admin.version_list') }}">Version Editor</a></li>
+		<li><a href="{{ url_for('admin.switch_user') }}">Sign in as another user</a></li>
 	</ul>
 
-	<div class="box box_grey">
-		<h2>Do action</h2>
+	<div class="card my-4">
+		<h2 class="card-header">Do action</h2>
 
-		<form method="post" action="" class="box-body">
+		<form method="post" action="" class="card-body">
 			<input type="hidden" name="csrf_token" value="{{ csrf_token() }}" />
 			<select name="action">
-				<option value="importusers" selected>Create users from mod list</option>
+				<option value="delstuckreleases" selected>Delete stuck releases</option>
+				<option value="checkreleases">Validate all Zip releases</option>
+				<option value="importmodlist">Import forum topics</option>
+				<option value="recalcscores">Recalculate package scores</option>
+				<option value="checkusers">Check forum users</option>
 				<option value="importscreenshots">Import screenshots from VCS</option>
+				<!-- <option value="importdepends">Import dependencies from downloads</option> -->
+				<!-- <option value="modprovides">Set provides to mod name</option> -->
+				<!-- <option value="vcsrelease">Create VCS releases</option> -->
 			</select>
 			<input type="submit" value="Perform" />
 		</form>
 	</div>
 
-	<div class="box box_grey">
-		<h2>Restore Package</h2>
+	<div class="card my-4">
+		<h2 class="card-header">Restore Package</h2>
 
-		<form method="post" action="" class="box-body">
+		<form method="post" action="" class="card-body">
 			<input type="hidden" name="csrf_token" value="{{ csrf_token() }}" />
 			<input type="hidden" name="action" value="restore" />
 			<select name="package">

app/templates/admin/tags/edit.html

@@ -0,0 +1,27 @@
+{% extends "base.html" %}
+
+{% block title %}
+	{% if tag %}
+		Edit {{ tag.title }}
+	{% else %}
+		New tag
+	{% endif %}
+{% endblock %}
+
+{% block content %}
+	<p>
+		<a href="{{ url_for('admin.tag_list') }}">Back to list</a> |
+		<a href="{{ url_for('admin.create_edit_tag') }}">New Tag</a>
+	</p>
+
+	{% from "macros/forms.html" import render_field, render_submit_field %}
+	<form method="POST" action="" enctype="multipart/form-data">
+		{{ form.hidden_tag() }}
+
+		{{ render_field(form.title) }}
+		{% if tag %}
+			{{ render_field(form.name) }}
+		{% endif %}
+		{{ render_submit_field(form.submit) }}
+	</form>
+{% endblock %}